[
https://issues.apache.org/jira/browse/NUTCH-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409225#comment-16409225
]
Sebastian Nagel commented on NUTCH-2541:
----------------------------------------
It's a known problem of protocol-httpclient - it does not work with non-ASCII
characters (protocol-http works), cf. [this post on the user mailing
list|https://mail-archives.apache.org/mod_mbox/nutch-user/201110.mbox/%3c423d069425e8e04f8cb763fd690bd3f40f6c3...@atlvmx03.turner.com%3E]
about {{https://es.wikipedia.org/wiki/Española}}. The plugin
protocol-httpclient needs to be updated (NUTCH-1086) in general.
For 1.x the best-way is to enable urlnormalizer-basic so that URLs are escaped:
{noformat}
echo -e
"https://es.wikipedia.org/wiki/Española\nhttp://agahi.safirak.com/ads/850/پیچ-بند-بادی-هفتیری-1800-دور-بادی-جیسون.html";
\
| nutch normalizerchecker -Dplugin.includes=urlnormalizer-basic -stdin
https://es.wikipedia.org/wiki/Espa%C3%B1ola
http://agahi.safirak.com/ads/850/%D9%BE%DB%8C%DA%86-%D8%A8%D9%86%D8%AF-%D8%A8%D8%A7%D8%AF%DB%8C-%D9%87%D9%81%D8%AA%DB%8C%D8%B1%DB%8C-1800-%D8%AF%D9%88%D8%B1-%D8%A8%D8%A7%D8%AF%DB%8C-%D8%AC%DB%8C%D8%B3%D9%88%D9%86.html
{noformat}
- this does not work for 2.x (I've opened NUTCH-2542 to fix this)
- parsechecker fails by escaping escaped URLs a second time (NUTCH-2145)
Until all this issues are fixed, the best work-around is to use protocol-http.
> Arabic characters in the URL path are not properly escaped by the
> protocol-httpclient plugin
> --------------------------------------------------------------------------------------------
>
> Key: NUTCH-2541
> URL: https://issues.apache.org/jira/browse/NUTCH-2541
> Project: Nutch
> Issue Type: Bug
> Components: plugin, protocol
> Affects Versions: 2.3.1, 1.14
> Reporter: Jorge Luis Betancourt Gonzalez
> Priority: Major
>
> As reported on [1]
> When trying to crawl some URLs with Arabic characters Nutch will complain due
> to an {{InvalidArgumentException}}. This happens because the HTTP client
> library is using internally the {{java.net.URI}} which does not support this
> characters unless they're properly escaped.
> [1]
> https://stackoverflow.com/questions/49379007/apache-nutch-2-3-1-fetcher-giving-invalid-uri-exception/49395225?noredirect=1#comment85798974_49395225
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
