Lewis John McGibbney created NUTCH-3246:
-------------------------------------------

             Summary: Add status counters to the remaining uninstrumented tools
                 Key: NUTCH-3246
                 URL: https://issues.apache.org/jira/browse/NUTCH-3246
             Project: Nutch
          Issue Type: Sub-task
            Reporter: Lewis John McGibbney


Issue 5 covers \{{LinkDb}}, \{{SegmentMerger}}, \{{CrawlDbReader}}, 
\{{ReadHostDb}}, and \{{CrawlDbMerger}}. A search of \{{src/java}} and 
\{{src/plugin}} for \{{getCounter}} shows these tools also have none. No plugin 
records a Hadoop counter.

* \{{org.apache.nutch.crawl.LinkDbMerger}}
* \{{org.apache.nutch.crawl.LinkDbReader}}
* \{{org.apache.nutch.tools.FreeGenerator}}
* \{{org.apache.nutch.scoring.webgraph.LinkRank}}
* \{{org.apache.nutch.scoring.webgraph.ScoreUpdater}}
* \{{org.apache.nutch.scoring.webgraph.NodeDumper}}
* \{{org.apache.nutch.scoring.webgraph.NodeReader}}
* \{{org.apache.nutch.scoring.webgraph.LinkDumper}}
* \{{org.apache.nutch.segment.SegmentReader}}
* \{{org.apache.nutch.util.ProtocolStatusStatistics}}
* \{{org.apache.nutch.util.CrawlCompletionStats}}
* \{{org.apache.nutch.tools.arc.ArcSegmentCreator}}

These are readers, mergers, dumpers, and webgraph tools. They are not the 
fetch, generate, parse, or index loop. \{{WebGraph}} itself already has 
\{{added_links_total}} and \{{removed_links_total}} and is not in this list.

h2. Proposal

For each tool, add \{{records_processed_total}} and \{{records_failed_total}} 
under a new \{{nutch_}} group declared in \{{NutchMetrics}}. Add 
\{{records_skipped_total}} only where the tool already has a skip path. Cache 
the references in \{{setup}} where the tool is a mapper or reducer.

Do not add \{{ErrorTracker}} here. Swallowed per-record failures in 
\{{WebGraph}}, \{{DomainStatistics}}, and \{{QueueFeeder}} are a separate 
issue, and those three classes already have other counters.

h2. Acceptance

* Each tool in the list emits its new counters from \{{NutchMetrics}} constants.
* A run over a small fixture increments \{{records_processed_total}} by the 
number of records handled.
* \{{WebGraph}}'s existing counters are unchanged.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to