[ https://issues.apache.org/jira/browse/NUTCH-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nguyen Manh Tien updated NUTCH-1686: ------------------------------------ Attachment: NUTCH-1686.patch > Optimize UpdateDb to load less field from Store > ----------------------------------------------- > > Key: NUTCH-1686 > URL: https://issues.apache.org/jira/browse/NUTCH-1686 > Project: Nutch > Issue Type: Improvement > Affects Versions: 2.3 > Reporter: Nguyen Manh Tien > Attachments: NUTCH-1686.patch > > > While running large crawl i found that updatedb run very slow, especially the > Map task which loading data from store. > We can't use filter by batchId to load less url due to bug in NUTCH-1679 so > we must always update the whole table. > After checking the field loaded in UpdateDbJob i found that it load many > fields from store (at least 15/25 field) which make updatedb slow > I think that UpdateDbJob only need to load few field: SCORE, OUTLINKS, > METADATA which is used to compute link score, distance that i think the main > purpose of this job. > The other fields is used to compute url schedule to parser and fetcher, we > can move code to Parser or Fetcher whithout loading much new field because > many field are generated from parser. WE can also use gora filter for Fetcher > or Parser so load new field is not a problem. > I also add new field SCOREMETA to WebPage to store CASH, and DISTANCE. It is > currently store in METADATA. field CASH is used in OPICScoring which is used > only in UpdateDB and distance is used only in Generator and Updater so move > both field two new Metadata field can prevent reading METADATA at Generator > and Updater, METADATA contains many data that is used only at Parser and > Indexer > So with new change > UpdateDb only load SCORE, SCOREMATA (CASH, DISTANCE), OUTLINKS, MAKERS: we > don't need to load big family Fetch and INLINKS. > Generator only load SCOREMETA (which is smaller than current METADATA) -- This message was sent by Atlassian JIRA (v6.1.5#6160)