krickert commented on PR #1164: URL: https://github.com/apache/opennlp/pull/1164#issuecomment-4939333346
Thanks, the ZWJ finding was a real bug. Addressed in d21612e5 (branch also rebased onto the current OPENNLP-1868 head): Blocking: - ZWJ sequences and U+FE0E: the pictographic scan is now sequence-aware. A mapped pictograph does not fold when the previous code point is a ZWJ, when the match (including an absorbed selector) is followed by a ZWJ, or when it is followed by U+FE0E. HEART ON FIRE and COUPLE WITH HEART pass through intact, U+263A U+FE0E stays text-presentation, and tests cover each case in plain and aligned form. The javadoc claims were rescoped to match. - Deprecated normalizer in the langdetect default chain: kept deliberately and now documented on both sides. Existing language detector models were trained with pictographs blanked; folding to ASCII emoticons would inject n-grams they never saw. The deprecation note states the retention, and the use site carries a SuppressWarnings with that rationale. - TermAnalyzer emoji fold coverage: added. EMOJI_FOLD layer on a pictograph token (length-changing, GRINNING FACE to :D), pass-through on plain tokens, and the Dimension.EMOJI_FOLD default normalizer wiring. Minor: - Longest-first audit now loops over both direction tables. - Dangling U+FE0F after bare-row pictographs: fixed generically. A trailing U+FE0F after any mapped pictograph is absorbed into the fold and the alignment maps it as part of the replaced block; tested with a supplementary-plane emoji plus selector. - emojiFold() vs emojiToEmoticon(): intentional. TermAnalyzer builder methods mirror Dimension names (caseFold, accentFold, emojiFold matches EMOJI_FOLD), while TextNormalizer names rungs by operation and exposes both directions, where a bare emojiFold() would not say which one. The asymmetry is the same one Dimension itself has: only the emoji-to-emoticon direction is a per-token dimension. - readResolve(): deferred to the package-wide singleton hardening batch, same call as on #1138; fixing only the two new classes would leave the package inconsistent. - Boxed map lookup per code point: fixed with first-code-point range bounds on each direction table, so the common no-match case is two integer compares. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
