[
https://issues.apache.org/jira/browse/NUTCH-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242219#comment-13242219
]
Markus Jelsma commented on NUTCH-1321:
--------------------------------------
...or, we could do a toUnicode for outlinks or directly in the fetcher. This
also makes sense because as ASCII these URL's are longer, sometimes much
longer. This can stir trouble for filters that, partly, rely on string length.
If both conversions are implemented in the fetcher or protocol library then we
don't have to worry about it, and have better logging in the fetcher!
> IDNNormalizer
> -------------
>
> Key: NUTCH-1321
> URL: https://issues.apache.org/jira/browse/NUTCH-1321
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
>
> Right now, IDN's are indexed as ASCII. An IDNNormalizer is to be used with an
> indexer so it will encode ASCII URL's to their proper unicode equivalant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
