[
https://issues.apache.org/jira/browse/NUTCH-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477364#comment-17477364
]
ASF GitHub Bot commented on NUTCH-2935:
---------------------------------------
sebastian-nagel merged pull request #723:
URL: https://github.com/apache/nutch/pull/723
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@nutch.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DeduplicationJob: failure on URLs with invalid percent encoding
> ---------------------------------------------------------------
>
> Key: NUTCH-2935
> URL: https://issues.apache.org/jira/browse/NUTCH-2935
> Project: Nutch
> Issue Type: Bug
> Components: crawldb
> Affects Versions: 1.18
> Reporter: Sebastian Nagel
> Assignee: Sebastian Nagel
> Priority: Major
> Fix For: 1.19
>
>
> The DeduplicationJob may fail with an IllegalArgumentException on invalid
> percent encodings in URLs:
> {noformat}
> 2021-11-25 04:36:41,747 INFO mapreduce.Job: Task Id :
> attempt_1637669672674_0018_r_000193_0, Status : FAILED
> Error: java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters
> in escape (%) pattern - Error at index 0 in: "YR"
> at java.base/java.net.URLDecoder.decode(URLDecoder.java:232)
> at java.base/java.net.URLDecoder.decode(URLDecoder.java:142)
> at
> org.apache.nutch.crawl.DeduplicationJob$DedupReducer.getDuplicate(DeduplicationJob.java:211)
> ...
> Exception in thread "main" java.lang.RuntimeException: Crawl job did not
> succeed, job status:FAILED, reason: Task failed
> task_1637669672674_0018_r_000193
> Job failed as tasks failed. failedMaps:0 failedReduces:1 killedMaps:0
> killedReduces: 0
> {noformat}
> The IllegalArgumentException should be caught, logged and if only one of the
> two URLs with duplicated content is invalid, it should be flagged as
> duplicate while the valid URL "survives".
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
