[
https://issues.apache.org/jira/browse/NUTCH-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131909#comment-18131909
]
ASF GitHub Bot commented on NUTCH-2455:
---------------------------------------
lewismc commented on PR #888:
URL: https://github.com/apache/nutch/pull/888#issuecomment-6401201509
Hi @sebastian-nagel, the note above gives the timings. This is how the
partition cache works.
## Why `getPartition` was on the profile
With HostDb **on**, the selector job uses `SelectorWithHostDb`. For a
CrawlDb row the key's hostname is empty, so `getPartition` calls
`URLPartitioner`. That normalizes the URL with
`URLNormalizers.SCOPE_PARTITION`, parses it, and hashes the host (or domain, or
IP). It does that once per CrawlDb record. On the 50M fixture that is about 50
URLs per host, and the default partition rules do not change the host, so those
50 calls return the same partition. On that profile
`SelectorWithHostDb.getPartition` was about 4% of CPU, almost all of it
`URLNormalizers.normalize`, and about 22% of allocation.
HostDb rows are separate. Their key carries the hostname, and `getPartition`
hashes that hostname with `partition.url.seed`. Those rows are not cached.
## What the cache stores
`SelectorWithHostDb` keeps a `HashMap` from raw host to the partition
`URLPartitioner` already returned.
The raw host is read from the URL bytes: find `://`, drop userinfo up to
`@`, take the host up to `/`, `?`, `#`, or `:`, keep an IPv6 `[...]` intact,
and lowercase it. `http://user:[email protected]:80/p2` and
`http://h00000.bench.invalid/p0` share the key `h00000.bench.invalid`. A URL
with no scheme or no host returns null from that scan. Those URLs call
`URLPartitioner` every time and are not stored.
On a CrawlDb row:
1. If this map task's reduce-task count changed, the map is cleared.
`setConf` clears it too.
2. If the raw host is already in the map, return the stored int.
3. Otherwise call `partitioner.getPartition(url, score, numReduceTasks)` and
store that int under the raw host.
The first URL of a host still pays for normalization. Later URLs for that
host reuse the int. The cache does not hash the host itself, so the stored
value is whatever `URLPartitioner` decided for that first URL, including
`partition.url.mode` and `partition.url.seed`.
`Selector`, the no-HostDb partitioner, is unchanged. It still calls
`URLPartitioner` for every URL, and its key is still `FloatWritable`.
## When the stored int is wrong
The cache assumes every later URL with the same raw host would get the same
partition from `URLPartitioner`. That holds for the default `SCOPE_PARTITION`
rules, which do not rewrite the host from the path.
`testHostDbPartitionCacheMatchesUrlPartitioner` checks the first call and the
cache hit against `URLPartitioner` for the fixture URL shapes: a second path,
`index.html`, a query, userinfo with a port, and a mixed-case host.
A custom normalizer that rewrites the host from the path breaks that
assumption. `http://example.com/a` and `http://example.com/b` share a cache
key. If the rule sends `/b` to a different host, the second URL still gets the
partition stored for `/a`.
> Use secondary sorting for memory-efficient HostDb integration in Generator
> --------------------------------------------------------------------------
>
> Key: NUTCH-2455
> URL: https://issues.apache.org/jira/browse/NUTCH-2455
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Affects Versions: 1.13
> Reporter: Markus Jelsma
> Assignee: Lewis John McGibbney
> Priority: Major
> Fix For: 1.24
>
> Attachments: NUTCH-2455.patch
>
>
> Citing Sebastian at NUTCH-2420:
> ??The correct solution would be to use <host,score> pairs as keys in the
> Selector job, with a partitioner and secondary sorting so that all keys with
> same host end up in the same call of the reducer. If values can also hold a
> HostDb entry and the sort comparator guarantees that the HostDb entry
> (entries if partitioned by domain or IP) comes in front of all CrawlDb
> entries. But that would be a substantial improvement...??
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
