[ https://issues.apache.org/jira/browse/NUTCH-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alparslan Avcı updated NUTCH-1674: ---------------------------------- Attachment: NUTCH-1674_3.patch > Use batchId filter to enable scan (GORA-119) for Fetch,Parse,Update,Index > ------------------------------------------------------------------------- > > Key: NUTCH-1674 > URL: https://issues.apache.org/jira/browse/NUTCH-1674 > Project: Nutch > Issue Type: Improvement > Affects Versions: 2.3 > Reporter: Nguyen Manh Tien > Fix For: 2.3 > > Attachments: NUTCH-1674.patch, NUTCH-1674_2.patch, NUTCH-1674_3.patch > > > Nutch always scan the whole crawldb in each phrase (generate, fetch, parse, > update, index). When crawldb is big, the time to scan is bigger than the > actual processing time. > We really need to skip records while scanning using GORA-119 for example we > can only get records belong to a specified batchId. > In my crawl the filter reduce the time to scan from 90 min to 30 min. -- This message was sent by Atlassian JIRA (v6.1.5#6160)