Lewis John McGibbney created NUTCH-3242:
-------------------------------------------
Summary: Add status counters to the five core jobs that have none
Key: NUTCH-3242
URL: https://issues.apache.org/jira/browse/NUTCH-3242
Project: Nutch
Issue Type: Sub-task
Reporter: Lewis John McGibbney
Five core jobs call \{{getCounter}} nowhere:
* \{{org.apache.nutch.crawl.LinkDb}}
* \{{org.apache.nutch.segment.SegmentMerger}}
* \{{org.apache.nutch.crawl.CrawlDbReader}}
* \{{org.apache.nutch.hostdb.ReadHostDb}}
* \{{org.apache.nutch.crawl.CrawlDbMerger}}
[NUTCH-3155|https://issues.apache.org/jira/browse/NUTCH-3155] adds
\{{ErrorTracker}} to \{{LinkDb.LinkDbMapper}}, \{{LinkDbFilter}},
\{{CrawlDbReader.CrawlDbStatMapper}}, \{{SegmentMerger.SegmentMergerMapper}},
\{{ReadHostDb.ReadHostDbMapper}}, and \{{UpdateHostDbReducer}}. It does not add
status counters. \{{UpdateHostDbReducer}} already has counters
(\{{total_hosts_total}}, \{{skipped_not_eligible_total}},
\{{url_limit_not_reached_total}}) and is not part of this issue.
\{{SegmentMerger}} already catches per record, at the two \{{catch
(Exception)}} blocks in its mapper. \{{CrawlDbMerger}}'s reducer writes the
merged datum and does not catch per record; its only catches are around job
submission.
Other tools also have no counters (LinkDbMerger, LinkDbReader, FreeGenerator,
the WebGraph tools, SegmentReader, and several dumpers). Those are a separate
issue. This issue is the five classes above.
h2. Proposal
Add these counters, declared in \{{NutchMetrics}}, under a new group per job
(\{{nutch_linkdb}}, \{{nutch_segment_merger}}, \{{nutch_crawldb_reader}},
\{{nutch_hostdb_reader}}, \{{nutch_crawldb_merger}}):
* \{{records_processed_total}}
* \{{records_skipped_total}}
* \{{records_failed_total}}
Cache the references in \{{setup}}, matching Generator and Injector.
Leave \{{ErrorTracker}} for the classes named in NUTCH-3155 to that issue. Do
not add \{{ErrorTracker}} to \{{CrawlDbMerger}}. Its reducer does not catch per
record.
h2. Acceptance
* Each of the five jobs emits the three status counters from \{{NutchMetrics}}
constants, with the references cached.
* A run over a small fixture increments \{{records_processed_total}} by the
number of records handled, and increments \{{records_skipped_total}} or
\{{records_failed_total}} only for records that were actually skipped or failed.
* \{{UpdateHostDbReducer}} is unchanged.
* No class named in NUTCH-3155 gains a second \{{ErrorTracker}} from this issue.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)