[
https://issues.apache.org/jira/browse/LUCENE-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir updated LUCENE-1343:
--------------------------------
Attachment: LUCENE-1343.patch
Attached is a patch that implements UTR#30 as a tailored unicode normalization
form.
Essentially it acts as a combined "Internationalized AsciiFoldingFilter" +
NFKC_CaseFold (Unicode Case Folding, Default Ignorable removal, and NFKC
normalization).
This is a nice alternative to just using ICUNormalizer2Filter in the case that
you want "fuzzy matching" (e.g. ignore diacritical marks).
The patch is large because it contains all the source data files necessary for
gennorm2 to regenerate the 41KB binary trie file... the java implementation is
trivial.
> A replacement for ISOLatin1AccentFilter that does a more thorough job of
> removing diacritical marks or non-spacing modifiers.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1343
> URL: https://issues.apache.org/jira/browse/LUCENE-1343
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Reporter: Robert Haschart
> Assignee: Robert Muir
> Priority: Minor
> Attachments: LUCENE-1343.patch, normalizer.jar, UnicodeCharUtil.java,
> UnicodeNormalizationFilter.java, UnicodeNormalizationFilterFactory.java
>
>
> The ISOLatin1AccentFilter takes Unicode characters that have diacritical
> marks and replaces them with a version of that character with the diacritical
> mark removed. For example é becomes e. However another equally valid way of
> representing an accented character in Unicode is to have the unaccented
> character followed by a non-spacing modifier character (like this: é )
> The ISOLatin1AccentFilter doesn't handle the accents in decomposed unicode
> characters at all. Additionally there are some instances where a word will
> contain what looks like an accented character, that is actually considered to
> be a separate unaccented character such as Ł but which to make searching
> easier you want to fold onto the latin1 lookalike version L .
> The UnicodeNormalizationFilter can filter out accents and diacritical marks
> whether they occur as composed characters or decomposed characters, it can
> also handle cases where as described above characters that look like they
> have diacritics (but don't) are to be folded onto the letter that they look
> like ( Ł -> L )
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
