Bug#776658: lintian: Memory consumption of harness and html_reports

On Fri, 30 Jan 2015 17:38:40 -0800 Russ Allbery wrote: > Niels Thykier writes: > > > The html_reports process itself consumes up to 2GB while processing > > templates. It is possible that there is nothing we can do about that > > as there *is* a lot of data

Bug#776658: lintian: Memory consumption of harness and html_reports

Minor update: I have written and merged a few more patches to further reduce the memory consumption of html_reports. Based on my local testing, these changes reduce the general case memory consumption of html_reports by ~25%. The work by reducing the cost of sharing data for two tags, so it is

Bug#776658: lintian: Memory consumption of harness and html_reports

On 2015-01-30 18:30, Niels Thykier wrote: Package: lintian Version: 2.5.30+deb8u3 Severity: important The reporting framework consumes a rather substantial amount of memory. [...] The html_reports process itself consumes up to 2GB while processing templates. It is possible that

Bug#776658: lintian: Memory consumption of harness and html_reports

On 2015-02-02 22:13, Niels Thykier wrote: I got a patch series to implement this (see also [BRANCH]). I have also managed to do a few tests on lilburn.debian.org with no issues. The commit messages are at the bare minimum - apologise for that. Review/remarks welcome. With the rewrite: *

Bug#776658: lintian: Memory consumption of harness and html_reports

On 2015-01-31 10:06, Niels Thykier wrote: [...] I have started a different approach (see [1] for WIP code). It is mostly a parallel track to your idea, so they can certainly co-exist. The goal of this approach is to: * Split harness into a simple coordinator * Remove the Lab as a

Bug#776658: lintian: Memory consumption of harness and html_reports

On 2015-01-31 02:38, Russ Allbery wrote: Niels Thykier ni...@thykier.net writes: The html_reports process itself consumes up to 2GB while processing templates. It is possible that there is nothing we can do about that as there *is* a lot of data in play. But even then, we can free it as

Bug#776658: lintian: Memory consumption of harness and html_reports

Package: lintian Version: 2.5.30+deb8u3 Severity: important The reporting framework consumes a rather substantial amount of memory. The harness process itself hogs ~1GB of RAM. This in itself is not concerning. However, it retains this usage even while running lintian and html_reports. For

Bug#776658: lintian: Memory consumption of harness and html_reports

Niels Thykier ni...@thykier.net writes: The html_reports process itself consumes up to 2GB while processing templates. It is possible that there is nothing we can do about that as there *is* a lot of data in play. But even then, we can free it as soon as possible (so we do not keep it while