[ https://issues.apache.org/jira/browse/NUTCH-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14327500#comment-14327500 ]
Alfonso Nishikawa edited comment on NUTCH-1679 at 2/19/15 2:45 PM: ------------------------------------------------------------------- Hi, just comment that the proposed patch will have a considerably impact on the performance, since {{store.get(TableUtil.reverseUrl(url))}} gets all the data. The patch seems to be a solution. I suggest use the #get(key,String[]) version, using a column like fetchTime or any that surely exists to increase the performance and fetch only one small column in each get. At the same time, I suggested the improvement of creating an #exists(key) method in GORA-411 that could benefit from any particular datastore feature. At the same time, if it is possible, check in the reduce if it is defined a batchId to avoid querying the datastore (one less network message :) ) was (Author: alfonso.nishikawa): Hi, just comment that the proposed patch will have a considerably impact on the performance, since {{store.get(TableUtil.reverseUrl(url))}} gets all the data. The patch seems to be a solution. I suggest use the #get(key,String[]) version, using a column like fetchTime or any that surely exists to increase the performance and fetch only one small column in each get. At the same time, I suggested the improvement of creating an #exists(key) method in GORA-411 that could benefit from the particular datastores features. > UpdateDb using batchId, link may override crawled page. > ------------------------------------------------------- > > Key: NUTCH-1679 > URL: https://issues.apache.org/jira/browse/NUTCH-1679 > Project: Nutch > Issue Type: Bug > Affects Versions: 2.2.1 > Reporter: Tien Nguyen Manh > Priority: Critical > Fix For: 2.3.1 > > Attachments: NUTCH-1679.patch > > > The problem is in Hbase store, not sure about other store. > Suppose at first crawl cycle we crawl link A, then get an outlink B. > In second cycle we crawl link B which also has a link point to A > In second updatedb we load only page B from store, and will add A as new link > because it doesn't know A already exist in store and will override A. > UpdateDb must be run without batchId or we must set additionsAllowed=false > Here are code for new page > page = new WebPage(); > schedule.initializeSchedule(url, page); > page.setStatus(CrawlStatus.STATUS_UNFETCHED); > try { > scoringFilters.initialScore(url, page); > } catch (ScoringFilterException e) { > page.setScore(0.0f); > } > new page will override old page status, score, fetchTime, fetchInterval, > retries, metadata[CASH_KEY] > - i think we can change something here so that new page will only update one > column for example 'link' and if it is really a new page, we can initialize > all above fields in generator > - or we add operator checkAndPut to store so when add new page we will check > if already exist first -- This message was sent by Atlassian JIRA (v6.3.4#6332)