[ https://issues.apache.org/jira/browse/NUTCH-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Otis Gospodnetic updated NUTCH-1674: ------------------------------------ Summary: Use batchId filter to enable scan (GORA-119) for Fetch,Parse,Update,Index (was: Use batchId filter enable scan (GORA-119) for Fetch,Parse,Update,Index) > Use batchId filter to enable scan (GORA-119) for Fetch,Parse,Update,Index > ------------------------------------------------------------------------- > > Key: NUTCH-1674 > URL: https://issues.apache.org/jira/browse/NUTCH-1674 > Project: Nutch > Issue Type: Improvement > Affects Versions: 2.3 > Reporter: Nguyen Manh Tien > Fix For: 2.3 > > Attachments: NUTCH-1674.patch > > > Nutch always scan the whole crawldb in each phrase (generate, fetch, parse, > update, index). When crawldb is big, the time to scan is bigger than the > actual processing time. > We really need to skip records while scanning using GORA-119 for example we > can only get records belong to a specified batchId. > In my crawl the filter reduce the time to scan from 90 min to 30 min. -- This message was sent by Atlassian JIRA (v6.1#6144)