lewismc commented on PR #888: URL: https://github.com/apache/nutch/pull/888#issuecomment-6398729043
Hi @sebastian-nagel previously with a large CrawlDb with HostDb turned off, you stated the branch generate was about 20% slower than master. This week I ran some comparative profiling to get a better understanding of performance with and without hostdb on `master` and `NUTCH-2455` branches. ## What I ran I used the same CrawlDb for both builds, local mode, Java 17, the default 4 GB heap. Three runs each. The time below is the median of the `Generator: finished` line. Master is `495ff8070`. The branch is `ee6eecd68`. I attached the [Grafana Pyroscope Java agent](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/java/) and sent profiles to Grafana Cloud Profiles (service `nutch.generator` on `profiles-prod-008`). Each CrawlDb was measured as twelve generates: `master` and `NUTCH-2455`, HostDb **off** and HostDb **on**, three times each. The elapsed time in the tables below is the median of those three, taken from the `Generator: finished` line. Every one of those twelve exited `0` (success). HostDb **off** is the February regression. HostDb **on** is the change this PR is for: master still loads the whole HostDb in each reducer, and the branch keeps only the hosts that arrive in that reducer. RSS in the tables is resident set size, the physical RAM the Nutch process held. The figure is the peak during that run. Local mode still forces one politeness partition, and the selector reducers run one after another in a single JVM, so that peak is one process. I started with that twelve-run set at 250,000 URLs. Wall-clock already matched. RSS on the no-HostDb path was noisy, so I scaled the CrawlDb to 1 million URLs and then to 50 million and repeated the same twelve generates at each size. @sebastian-nagel does this get anywhere close to your "large" CrawlDb? **1,000,000 URLs, 200,000 hosts** | Path | Master elapsed | NUTCH-2455 elapsed | Master peak RSS | NUTCH-2455 peak RSS | | --- | ---: | ---: | ---: | ---: | | No HostDb | 17.6 s (17645 ms) | 17.7 s (17685 ms, +0.2%) | 846 MiB | 769 MiB | | HostDb | 23.7 s (23677 ms) | 18.6 s (18636 ms, −21%) | 1198 MiB | 793 MiB (−34%) | **50,000,000 URLs, 1,000,000 hosts** | Path | Master elapsed | NUTCH-2455 elapsed | Master peak RSS | NUTCH-2455 peak RSS | | --- | ---: | ---: | ---: | ---: | | No HostDb | 7 min 34 s (454454 ms) | 7 min 36 s (455533 ms, +0.2%) | 4024 MiB | 3975 MiB (−1%) | | HostDb | 8 min 14 s (493666 ms) | 7 min 51 s (471043 ms, −4.6%) | 4211 MiB | 3935 MiB (−6.6%) | The branch log with HostDb **off** says `using standard generation (no HostDb)`, which is the `FloatWritable` job. With HostDb **on** it says `using HostDb integration with secondary sorting`. At 50 million URLs every run sits on the 4 GB heap because the sort dominates, so the HostDb gap is smaller than at 1 million URLs. These numbers leave Generator as it is on the branch. If you'd like @sebastian-nagel (or any other committer) I can invite you to my Grafana Cloud stack so you can view and compare the profiles and we can take the investigation further. Please let me know. Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
