[ https://issues.apache.org/jira/browse/NUTCH-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808006#comment-13808006 ]
Talat UYARER commented on NUTCH-1643: ------------------------------------- Hi [~lewismc], I can look every protocol for this improvement. But we should consider your second item. I look at ParseUtil.java when parsers return null, it writes warning log. I think this is not a problem. What do you think ? I will uplaod for other protocols. I added parse methods from ParseUtil.java: {code:title=ParseUtil.java|borderStyle=solid} public Parse parse(String url, WebPage page) throws ParserNotFound, ParseException { Parser[] parsers = null; String contentType = TableUtil.toString(page.getContentType()); parsers = this.parserFactory.getParsers(contentType, url); for (int i=0; i<parsers.length; i++) { if (LOG.isDebugEnabled()) { LOG.debug("Parsing [" + url + "] with [" + parsers[i] + "]"); } Parse parse = null; if (maxParseTime!=-1) parse = runParser(parsers[i], url, page); else parse = parsers[i].getParse(url, page); if (parse!=null && ParseStatusUtils.isSuccess(parse.getParseStatus())) { return parse; } } LOG.warn("Unable to successfully parse content " + url + " of type " + contentType); return ParseStatusUtils.getEmptyParse(new ParseException("Unable to successfully parse content"), null); } {code} > Unnecessary fetching with http.content.limit when using protocol-http > --------------------------------------------------------------------- > > Key: NUTCH-1643 > URL: https://issues.apache.org/jira/browse/NUTCH-1643 > Project: Nutch > Issue Type: Bug > Components: protocol > Affects Versions: 2.1, 2.2, 2.2.1 > Reporter: Talat UYARER > Priority: Minor > Fix For: 2.3 > > Attachments: NUTCH-1643.patch, NUTCH-1643v2.patch > > > In protocol-http, Even If I have http.content.limit value set, protocol-http > fetches files of all sizes (larger files are fetched until limit allows). > But when Parsing, parser skips incomplete files (if parser.skip.truncated > configuration is true). It seems like an unnecessary effort to partially > fetch contents larger than limit if they are not gonna be parsed. -- This message was sent by Atlassian JIRA (v6.1#6144)