[ https://issues.apache.org/jira/browse/NUTCH-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833620#comment-13833620 ]
Lewis John McGibbney commented on NUTCH-1674: --------------------------------------------- Hi [~otis] {bq} what do you mean by consolidate here? I mean lets get GORA-119 in to the codebase first based on gora-hbase users & committers comments then we can do a release. I'm looking forward to pushing this out. Thanks for the work on these important issues. > Use batchId filter to enable scan (GORA-119) for Fetch,Parse,Update,Index > ------------------------------------------------------------------------- > > Key: NUTCH-1674 > URL: https://issues.apache.org/jira/browse/NUTCH-1674 > Project: Nutch > Issue Type: Improvement > Affects Versions: 2.3 > Reporter: Nguyen Manh Tien > Fix For: 2.3 > > Attachments: NUTCH-1674.patch > > > Nutch always scan the whole crawldb in each phrase (generate, fetch, parse, > update, index). When crawldb is big, the time to scan is bigger than the > actual processing time. > We really need to skip records while scanning using GORA-119 for example we > can only get records belong to a specified batchId. > In my crawl the filter reduce the time to scan from 90 min to 30 min. -- This message was sent by Atlassian JIRA (v6.1#6144)