On 07/06/2013, at 5:52 PM, Szczepan Faber <szczepan.fa...@gradleware.com> wrote:
> >@Szczepan, this would be a good candidate to add to your list: reusing > >warmed up user classes (which includes scripts and plugins) should give a > >nice performance improvement to build time, and configuration time in > >particular, with the daemon > > Is there a chance it also tangibly reduces the heap consumption? Absolutely. This is why I think it might be a good option for you to look at. > How big this story is? It should be relatively quick to spike, to see if it's worthwhile. Implementation is another story. I think the bulk of the work would be detangling some of the file snapshot and change detection stuff from task up-to-date checking so that we can reuse it for classpath up-to-date checking. > Say that I'm interested in a tangible performance improvement (+ perhaps > heap), without yet solving those other interesting use cases. We wouldn't need to solve the other use cases. > > Cheers! > > > On Fri, Jun 7, 2013 at 1:19 AM, Adam Murdoch <adam.murd...@gradleware.com> > wrote: > > On 05/06/2013, at 3:38 PM, Luke Daley <luke.da...@gradle.biz> wrote: > >> >> >> On 05/06/2013, at 2:37, Adam Murdoch <adam.murd...@gradleware.com> wrote: >> >>> >>> On 05/06/2013, at 3:57 AM, Luke Daley <luke.da...@gradleware.com> wrote: >>> >>>> Hi, >>>> >>>> I just tracked down a memory leak in the Artifactory plugin. I've raised >>>> with them to fix this, but this is a symptom of a more general leak we >>>> have in place… I think. >>>> >>>> Groovy keeps a global registry of metaclass of all loaded classes that get >>>> touched by Groovy. We load Groovy in a persistent classloader across >>>> builds. This means that build level user classes (e.g. non core plugins) >>>> end up getting loaded into Groovy's metaclass registry each time there is >>>> a build. This means we are always leaking permgen, and that we are very >>>> susceptible to bad leaks if user code keeps a static state reference to >>>> something, as was the case with the artifactory plugin. They ended up >>>> holding on to a reference of the root project statically (through Groovy >>>> meta programming) which means a lot was leaked. >>>> >>>> We can't really change the classloader structure to unload the whole meta >>>> class registry between builds as that would defeat the main purpose of the >>>> daemon: to load Groovy once. >>> >>> The purpose of the daemon is to keep the things that a build needs >>> warmed-up. This includes groovy, but it also includes the user classes used >>> by the build (and other state, too). So, we only want to discard the >>> classes and associated meta-data that we think are unlikely to be needed >>> any more - classes from class loaders that we discard because their >>> classpath has changed or class loaders that we discard because we haven't >>> run the associated build for a while. >> >> My read of things was that we don't cache any user code at the moment. The >> growing meta class registry seems to indicate that we are dumping user >> classes between builds. > > I got confused. We aren't reusing anything. For some reason I thought we had > implemented this. This would be a good place to start as far as fixing the > leak goes. > > @Szczepan, this would be a good candidate to add to your list: reusing warmed > up user classes (which includes scripts and plugins) should give a nice > performance improvement to build time, and configuration time in particular, > with the daemon. With the daemon, we see roughly a 20% improvement in > execution time between the 2nd and 5th (or so) build invocations as the > hotspot compiler does its thing. This is beyond the initial improvement > between the 1st and 2nd invocations. I'd expect to see something similar for > user code. > > It also reduces the chance of memory leaks. It eliminates some types of > problems and adds some others, but I think on the balance it should be better. > > Reusing the user classes means we'd need to track changes to classpaths > efficiently, and this means we can do some interesting things on change: > - invalidate a task's outputs when the task implementation changes > - notify the tooling API client that the model may have changed. > > And we can cache some interesting information derived from a classpath: > - the API for the classpath, which can be used for up-to-date checks > and compilation. > - plugin meta-data scanned from the classpath (eg > @Plugin(id='my-plugin') annotation, or plugin supplied services). > - test detection. > - the project model, or enough of the model to decide it the project's > outputs are up-to-date or not. > > > -- > Adam Murdoch > Gradle Co-founder > http://www.gradle.org > VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting > http://www.gradleware.com > > Join us at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: > http://www.gradlesummit.com > > > > > -- > Szczepan Faber > Principal engineer@gradleware; Lead@mockito > Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: > http://www.gradlesummit.com -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com Join us at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: http://www.gradlesummit.com
