[ https://issues.apache.org/jira/browse/NUTCH-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881202#comment-13881202 ]
Markus Jelsma commented on NUTCH-1715: -------------------------------------- Great work! > RobotRulesParser adds additional '*' to the robots name > ------------------------------------------------------- > > Key: NUTCH-1715 > URL: https://issues.apache.org/jira/browse/NUTCH-1715 > Project: Nutch > Issue Type: Bug > Components: fetcher > Affects Versions: 1.7, 2.2.1 > Reporter: Tejas Patil > Assignee: Tejas Patil > Fix For: 2.3, 1.8 > > Attachments: NUTCH-1715.2.x.patch, NUTCH-1715.trunk.patch > > > In RobotRulesParser, when Nutch creates a agent string from multiple agents, > it combines agents from both 'http.agent.name' and 'http.robots.agents'. > Along with that it appends a wildcard (ie. *) to it in the end. This is sent > to crawler commons while parsing the rules. The wildcard gets matched first > in robots file with (User-agent: *) if that comes before any other matching > rule thus resulting in a allowed url being robots denied. > This issue was reported by [~markus17]. The discussion over nutch-user is > here: > http://mail-archives.apache.org/mod_mbox/nutch-user/201401.mbox/%3CCAFKhtFzBRpVv4MULSxw8RDRR_wbivOt%3DnhFX-w621BR8q%2BxVDQ%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.1.5#6160)