Lewis John McGibbney created NUTCH-3246:
-------------------------------------------
Summary: Add status counters to the remaining uninstrumented tools
Key: NUTCH-3246
URL: https://issues.apache.org/jira/browse/NUTCH-3246
Project: Nutch
Issue Type: Sub-task
Reporter: Lewis John McGibbney
Issue 5 covers \{{LinkDb}}, \{{SegmentMerger}}, \{{CrawlDbReader}},
\{{ReadHostDb}}, and \{{CrawlDbMerger}}. A search of \{{src/java}} and
\{{src/plugin}} for \{{getCounter}} shows these tools also have none. No plugin
records a Hadoop counter.
* \{{org.apache.nutch.crawl.LinkDbMerger}}
* \{{org.apache.nutch.crawl.LinkDbReader}}
* \{{org.apache.nutch.tools.FreeGenerator}}
* \{{org.apache.nutch.scoring.webgraph.LinkRank}}
* \{{org.apache.nutch.scoring.webgraph.ScoreUpdater}}
* \{{org.apache.nutch.scoring.webgraph.NodeDumper}}
* \{{org.apache.nutch.scoring.webgraph.NodeReader}}
* \{{org.apache.nutch.scoring.webgraph.LinkDumper}}
* \{{org.apache.nutch.segment.SegmentReader}}
* \{{org.apache.nutch.util.ProtocolStatusStatistics}}
* \{{org.apache.nutch.util.CrawlCompletionStats}}
* \{{org.apache.nutch.tools.arc.ArcSegmentCreator}}
These are readers, mergers, dumpers, and webgraph tools. They are not the
fetch, generate, parse, or index loop. \{{WebGraph}} itself already has
\{{added_links_total}} and \{{removed_links_total}} and is not in this list.
h2. Proposal
For each tool, add \{{records_processed_total}} and \{{records_failed_total}}
under a new \{{nutch_}} group declared in \{{NutchMetrics}}. Add
\{{records_skipped_total}} only where the tool already has a skip path. Cache
the references in \{{setup}} where the tool is a mapper or reducer.
Do not add \{{ErrorTracker}} here. Swallowed per-record failures in
\{{WebGraph}}, \{{DomainStatistics}}, and \{{QueueFeeder}} are a separate
issue, and those three classes already have other counters.
h2. Acceptance
* Each tool in the list emits its new counters from \{{NutchMetrics}} constants.
* A run over a small fixture increments \{{records_processed_total}} by the
number of records handled.
* \{{WebGraph}}'s existing counters are unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)