[
https://issues.apache.org/jira/browse/OPENNLP-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118102#comment-18118102
]
ASF GitHub Bot commented on OPENNLP-1974:
-----------------------------------------
rzo1 commented on PR #1313:
URL: https://github.com/apache/opennlp/pull/1313#issuecomment-5781673039
@krickert I would treat a version update 17 -> 18 similar to a "code"
dependency version update, so it is trackable in the release notes and doesn't
hide in a normalizer ticket :)
> RegEx removal part 8: EmojiCharSequenceNormalizer blanks hyphens and BMP
> characters
> -----------------------------------------------------------------------------------
>
> Key: OPENNLP-1974
> URL: https://issues.apache.org/jira/browse/OPENNLP-1974
> Project: OpenNLP
> Issue Type: Bug
> Reporter: Richard Zowalla
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1973.
> The pattern of {{EmojiCharSequenceNormalizer}} was parsed as the range
> U\+D83C to U\+10FC00 plus a literal hyphen, so it also blanked the hyphen and
> the BMP characters from U\+D83C to U\+FFFF \(fullwidth and halfwidth forms,
> CJK compatibility ideographs, Arabic presentation forms, the private use
> area, U\+FE0F, unpaired surrogates\) and kept U\+10FC01 to U\+10FFFF.
> The normalizer now replaces each run of supplementary code points \(U\+10000
> and above\) with one space, using a code point scan instead of the regular
> expression. It is part of the default language detector chain, so models
> trained with an earlier release see different input; under
> {{\-Dopennlp.compat.mode=LEGACY}} the earlier output is produced.
> Before merge: language detector eval build.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
