[
https://issues.apache.org/jira/browse/NUTCH-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131870#comment-18131870
]
ASF GitHub Bot commented on NUTCH-2455:
---------------------------------------
lewismc commented on PR #888:
URL: https://github.com/apache/nutch/pull/888#issuecomment-6398729043
Hi @sebastian-nagel previously with a large CrawlDb with HostDb turned off,
you stated the branch generate was about 20% slower than master. This week I
ran some comparative profiling to get a better understanding of performance
with and without hostdb on `master` and `NUTCH-2455` branches.
## What I ran
I used the same CrawlDb for both builds, local mode, Java 17, the default 4
GB heap. Three runs each. The time below is the median of the `Generator:
finished` line. Master is `495ff8070`. The branch is `ee6eecd68`. I attached
the [Grafana Pyroscope Java
agent](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/java/)
and sent profiles to Grafana Cloud Profiles (service `nutch.generator` on
`profiles-prod-008`).
Each CrawlDb was measured as twelve generates: `master` and `NUTCH-2455`,
HostDb **off** and HostDb **on**, three times each. The elapsed time in the
tables below is the median of those three, taken from the `Generator: finished`
line. Every one of those twelve exited `0` (success).
HostDb **off** is the February regression. HostDb **on** is the change this
PR is for: master still loads the whole HostDb in each reducer, and the branch
keeps only the hosts that arrive in that reducer.
RSS in the tables is resident set size, the physical RAM the Nutch process
held. The figure is the peak during that run. Local mode still forces one
politeness partition, and the selector reducers run one after another in a
single JVM, so that peak is one process.
I started with that twelve-run set at 250,000 URLs. Wall-clock already
matched. RSS on the no-HostDb path was noisy, so I scaled the CrawlDb to 1
million URLs and then to 50 million and repeated the same twelve generates at
each size. @sebastian-nagel does this get anywhere close to your "large"
CrawlDb?
**1,000,000 URLs, 200,000 hosts**
| Path | Master elapsed | NUTCH-2455 elapsed | Master peak RSS | NUTCH-2455
peak RSS |
| -
> Use secondary sorting for memory-efficient HostDb integration in Generator
> --------------------------------------------------------------------------
>
> Key: NUTCH-2455
> URL: https://issues.apache.org/jira/browse/NUTCH-2455
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Affects Versions: 1.13
> Reporter: Markus Jelsma
> Assignee: Lewis John McGibbney
> Priority: Major
> Fix For: 1.24
>
> Attachments: NUTCH-2455.patch
>
>
> Citing Sebastian at NUTCH-2420:
> ??The correct solution would be to use <host,score> pairs as keys in the
> Selector job, with a partitioner and secondary sorting so that all keys with
> same host end up in the same call of the reducer. If values can also hold a
> HostDb entry and the sort comparator guarantees that the HostDb entry
> (entries if partitioned by domain or IP) comes in front of all CrawlDb
> entries. But that would be a substantial improvement...??
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
