[
https://issues.apache.org/jira/browse/NUTCH-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lewis John McGibbney updated NUTCH-3150:
----------------------------------------
Description:
This issue expands on initial work to cache counters in hot paths only. See
NUTCH-3141.
After some analysis this issue can be scoped as follows
9 files to modify (~45 counters total):
* Fetcher.java - 5 counters (main fetch loop)
* ResolverThread.java - 10 counters (DNS resolution)
* Generator.java - 9 counters (URL generation)
* CrawlDbReducer.java - Dynamic counters with Map
* Injector.java - 6 counters (mapper + reducer)
* CrawlDbFilter.java - 3 counters
* WebGraph.java - 2 counters
* DeduplicationJob.java - 1 counter
* CleaningJob.java - 1 counter
was:
This issue expands on initial work to cache counters in hot paths only. See
NUTCH-3141.
After some analysis this issue can be scoped as follows
9 files to modify (~45 counters total):
* Fetcher.java - 5 counters (main fetch loop)
* ResolverThread.java - 10 counters (DNS resolution)
* Generator.java - 9 counters (URL generation)
* CrawlDbReducer.java - Dynamic counters with Map
* Injector.java - 6 counters (mapper + reducer)
* CrawlDbFilter.java - 3 counters
* WebGraph.java - 2 counters
* DeduplicationJob.java - 1 counter
* CleaningJob.java - 1 counter
> Expand Caching Hadoop Counter References
> ----------------------------------------
>
> Key: NUTCH-3150
> URL: https://issues.apache.org/jira/browse/NUTCH-3150
> Project: Nutch
> Issue Type: Sub-task
> Components: metrics
> Reporter: Lewis John McGibbney
> Assignee: Lewis John McGibbney
> Priority: Major
> Fix For: 1.22
>
>
> This issue expands on initial work to cache counters in hot paths only. See
> NUTCH-3141.
> After some analysis this issue can be scoped as follows
> 9 files to modify (~45 counters total):
> * Fetcher.java - 5 counters (main fetch loop)
> * ResolverThread.java - 10 counters (DNS resolution)
> * Generator.java - 9 counters (URL generation)
> * CrawlDbReducer.java - Dynamic counters with Map
> * Injector.java - 6 counters (mapper + reducer)
> * CrawlDbFilter.java - 3 counters
> * WebGraph.java - 2 counters
> * DeduplicationJob.java - 1 counter
> * CleaningJob.java - 1 counter
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
