[
https://issues.apache.org/jira/browse/OPENNLP-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121045#comment-18121045
]
ASF GitHub Bot commented on OPENNLP-1974:
-----------------------------------------
krickert commented on PR #1313:
URL: https://github.com/apache/opennlp/pull/1313#issuecomment-5908561380
Pushed the fixes and moved the data back to Emoji 17.0. The Emoji 18.0
update is tracked in #1347 for 3.1.
LEGACY mode keeps the old behavior without using regex. The tests compare
its output against the old pattern.
Removed the deprecation and added the normalizer to the default chain. Emoji
next to stray components are preserved; other complete emoji are removed,
including one extra U+FE0F after them.
Updated the Javadoc and Unicode license notices, and moved the tests to
opennlp-runtime.
Eval:
https://ci-builds.apache.org/job/OpenNLP/job/eval-tests-configurable/91/console
> RegEx removal part 8: EmojiCharSequenceNormalizer blanks hyphens and BMP
> characters
> -----------------------------------------------------------------------------------
>
> Key: OPENNLP-1974
> URL: https://issues.apache.org/jira/browse/OPENNLP-1974
> Project: OpenNLP
> Issue Type: Bug
> Reporter: Richard Zowalla
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1973.
> The pattern of {{EmojiCharSequenceNormalizer}} was parsed as the range
> U\+D83C to U\+10FC00 plus a literal hyphen, so it also blanked the hyphen and
> the BMP characters from U\+D83C to U\+FFFF \(fullwidth and halfwidth forms,
> CJK compatibility ideographs, Arabic presentation forms, the private use
> area, U\+FE0F, unpaired surrogates\) and kept U\+10FC01 to U\+10FFFF.
> The normalizer now replaces each run of supplementary code points \(U\+10000
> and above\) with one space, using a code point scan instead of the regular
> expression. It is part of the default language detector chain, so models
> trained with an earlier release see different input; under
> {{\-Dopennlp.compat.mode=LEGACY}} the earlier output is produced.
> Before merge: language detector eval build.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
