[
https://issues.apache.org/jira/browse/NUTCH-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131906#comment-18131906
]
ASF GitHub Bot commented on NUTCH-2455:
---------------------------------------
lewismc commented on PR #888:
URL: https://github.com/apache/nutch/pull/888#issuecomment-6401157672
Hi @sebastian-nagel, a follow-up on the 50M HostDb profile from the runs
above.
## What a updated profiling exercise showed
On each 50M HostDb run, almost all of the CPU was local filesystem I/O in
the spill and merge. Inside Generator, `SelectorWithHostDb.getPartition` was
about 4% of CPU. Nearly all of that was `URLNormalizers.normalize`, and that
same frame was a large share of allocation. `MapOutputBuffer.compare` was about
1.3%. A raw-byte `ScoreHostKeyComparator` would not have moved the 50M time, so
I left the comparator as it is.
The CrawlDb side was calling `URLPartitioner` once per URL. On this fixture
that is about 50 URLs per host, and the default normalizers do not change the
host. The new code keeps a host-to-partition map on `SelectorWithHostDb`. The
first CrawlDb URL for a host still goes through `URLPartitioner`. Later URLs
for that same raw host reuse the partition. HostDb rows still partition by
hostname hash. The no-HostDb job is still the `FloatWritable` selector.
For the new run, the profiles are on the same `nutch.generator` service,
labeled `git_sha=6d9a64e63-hostcache`. In that profile, `getPartition` and
`URLNormalizers.normalize` no longer show up in the summary. The reducer is
still mostly I/O.
## Same 4 GB setup
I performed a new run with the same CrawlDbs, local mode, Java 17, and the
default 4 GB heap as the tables above. Three runs, median of `Generator:
finished`. Every run exited 0. I did not change the heap. The comparison point
is the NUTCH-2455 column already posted (`ee6eecd68`).
**1,000,000 URLs, 200,000 hosts**
| Path | Previous NUTCH-2455 | With the partition cache | Peak RSS |
| -
> Use secondary sorting for memory-efficient HostDb integration in Generator
> --------------------------------------------------------------------------
>
> Key: NUTCH-2455
> URL: https://issues.apache.org/jira/browse/NUTCH-2455
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Affects Versions: 1.13
> Reporter: Markus Jelsma
> Assignee: Lewis John McGibbney
> Priority: Major
> Fix For: 1.24
>
> Attachments: NUTCH-2455.patch
>
>
> Citing Sebastian at NUTCH-2420:
> ??The correct solution would be to use <host,score> pairs as keys in the
> Selector job, with a partitioner and secondary sorting so that all keys with
> same host end up in the same call of the reducer. If values can also hold a
> HostDb entry and the sort comparator guarantees that the HostDb entry
> (entries if partitioned by domain or IP) comes in front of all CrawlDb
> entries. But that would be a substantial improvement...??
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
