[ https://issues.apache.org/jira/browse/NUTCH-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662728#comment-13662728 ]
Hudson commented on NUTCH-1513: ------------------------------- Integrated in Nutch-nutchgora #613 (See [https://builds.apache.org/job/Nutch-nutchgora/613/]) NUTCH-1513 Support Robots.txt for Ftp urls (Revision 1484637) Result = SUCCESS tejasp : http://svn.apache.org/viewvc/nutch/branches/2.x/?view=rev&rev=1484637 Files : * /nutch/branches/2.x/CHANGES.txt * /nutch/branches/2.x/src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/Ftp.java * /nutch/branches/2.x/src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/FtpRobotRulesParser.java > Support Robots.txt for Ftp urls > ------------------------------- > > Key: NUTCH-1513 > URL: https://issues.apache.org/jira/browse/NUTCH-1513 > Project: Nutch > Issue Type: Improvement > Affects Versions: 1.7, 2.2 > Reporter: Tejas Patil > Assignee: Tejas Patil > Priority: Minor > Labels: robots.txt > Fix For: 2.3, 1.8 > > Attachments: NUTCH-1513.2.x.v2.patch, NUTCH-1513.trunk.patch, > NUTCH-1513.trunk.v2.patch > > > As per [0], a FTP website can have robots.txt like [1]. In the nutch code, > Ftp plugin is not parsing the robots file and accepting all urls. > In "_src/plugin/protocol-ftp/src/java/org/apache/nutch/protocol/ftp/Ftp.java_" > {noformat} public RobotRules getRobotRules(Text url, CrawlDatum datum) { > return EmptyRobotRules.RULES; > }{noformat} > Its not clear of this was part of design or if its a bug. > [0] : > https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt > [1] : ftp://example.com/robots.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira