>@Szczepan, this would be a good candidate to add to your list: reusing warmed up user classes (which includes scripts and plugins) should give a nice performance improvement to build time, and configuration time in particular, with the daemon
Is there a chance it also tangibly reduces the heap consumption? How big this story is? Say that I'm interested in a tangible performance improvement (+ perhaps heap), without yet solving those other interesting use cases. Cheers! On Fri, Jun 7, 2013 at 1:19 AM, Adam Murdoch <adam.murd...@gradleware.com>wrote: > > On 05/06/2013, at 3:38 PM, Luke Daley <luke.da...@gradle.biz> wrote: > > > > On 05/06/2013, at 2:37, Adam Murdoch <adam.murd...@gradleware.com> wrote: > > > On 05/06/2013, at 3:57 AM, Luke Daley <luke.da...@gradleware.com> wrote: > > Hi, > > I just tracked down a memory leak in the Artifactory plugin. I've raised > with them to fix this, but this is a symptom of a more general leak we have > in place… I think. > > Groovy keeps a global registry of metaclass of all loaded classes that get > touched by Groovy. We load Groovy in a persistent classloader across > builds. This means that build level user classes (e.g. non core plugins) > end up getting loaded into Groovy's metaclass registry each time there is a > build. This means we are always leaking permgen, and that we are very > susceptible to bad leaks if user code keeps a static state reference to > something, as was the case with the artifactory plugin. They ended up > holding on to a reference of the root project statically (through Groovy > meta programming) which means a lot was leaked. > > We can't really change the classloader structure to unload the whole meta > class registry between builds as that would defeat the main purpose of the > daemon: to load Groovy once. > > > The purpose of the daemon is to keep the things that a build needs > warmed-up. This includes groovy, but it also includes the user classes used > by the build (and other state, too). So, we only want to discard the > classes and associated meta-data that we think are unlikely to be needed > any more - classes from class loaders that we discard because their > classpath has changed or class loaders that we discard because we haven't > run the associated build for a while. > > > My read of things was that we don't cache any user code at the moment. The > growing meta class registry seems to indicate that we are dumping user > classes between builds. > > > I got confused. We aren't reusing anything. For some reason I thought we > had implemented this. This would be a good place to start as far as fixing > the leak goes. > > @Szczepan, this would be a good candidate to add to your list: reusing > warmed up user classes (which includes scripts and plugins) should give a > nice performance improvement to build time, and configuration time in > particular, with the daemon. With the daemon, we see roughly a 20% > improvement in execution time between the 2nd and 5th (or so) build > invocations as the hotspot compiler does its thing. This is beyond the > initial improvement between the 1st and 2nd invocations. I'd expect to see > something similar for user code. > > It also reduces the chance of memory leaks. It eliminates some types of > problems and adds some others, but I think on the balance it should be > better. > > Reusing the user classes means we'd need to track changes to classpaths > efficiently, and this means we can do some interesting things on change: > - invalidate a task's outputs when the task implementation changes > - notify the tooling API client that the model may have changed. > > And we can cache some interesting information derived from a classpath: > - the API for the classpath, which can be used for up-to-date checks and > compilation. > - plugin meta-data scanned from the classpath (eg @Plugin(id='my-plugin') > annotation, or plugin supplied services). > - test detection. > - the project model, or enough of the model to decide it the project's > outputs are up-to-date or not. > > > -- > Adam Murdoch > Gradle Co-founder > http://www.gradle.org > VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting > http://www.gradleware.com > > Join us at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: > http://www.gradlesummit.com > > -- Szczepan Faber Principal engineer@gradleware; Lead@mockito Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: http://www.gradlesummit.com
