On 05/06/2013, at 3:38 PM, Luke Daley <luke.da...@gradle.biz> wrote:
>
>
> On 05/06/2013, at 2:37, Adam Murdoch <adam.murd...@gradleware.com> wrote:
>
>>
>> On 05/06/2013, at 3:57 AM, Luke Daley <luke.da...@gradleware.com> wrote:
>>
>>> Hi,
>>>
>>> I just tracked down a memory leak in the Artifactory plugin. I've raised
>>> with them to fix this, but this is a symptom of a more general leak we have
>>> in place… I think.
>>>
>>> Groovy keeps a global registry of metaclass of all loaded classes that get
>>> touched by Groovy. We load Groovy in a persistent classloader across
>>> builds. This means that build level user classes (e.g. non core plugins)
>>> end up getting loaded into Groovy's metaclass registry each time there is a
>>> build. This means we are always leaking permgen, and that we are very
>>> susceptible to bad leaks if user code keeps a static state reference to
>>> something, as was the case with the artifactory plugin. They ended up
>>> holding on to a reference of the root project statically (through Groovy
>>> meta programming) which means a lot was leaked.
>>>
>>> We can't really change the classloader structure to unload the whole meta
>>> class registry between builds as that would defeat the main purpose of the
>>> daemon: to load Groovy once.
>>
>> The purpose of the daemon is to keep the things that a build needs
>> warmed-up. This includes groovy, but it also includes the user classes used
>> by the build (and other state, too). So, we only want to discard the classes
>> and associated meta-data that we think are unlikely to be needed any more -
>> classes from class loaders that we discard because their classpath has
>> changed or class loaders that we discard because we haven't run the
>> associated build for a while.
>
> My read of things was that we don't cache any user code at the moment. The
> growing meta class registry seems to indicate that we are dumping user
> classes between builds.
I got confused. We aren't reusing anything. For some reason I thought we had
implemented this. This would be a good place to start as far as fixing the leak
goes.
@Szczepan, this would be a good candidate to add to your list: reusing warmed
up user classes (which includes scripts and plugins) should give a nice
performance improvement to build time, and configuration time in particular,
with the daemon. With the daemon, we see roughly a 20% improvement in execution
time between the 2nd and 5th (or so) build invocations as the hotspot compiler
does its thing. This is beyond the initial improvement between the 1st and 2nd
invocations. I'd expect to see something similar for user code.
It also reduces the chance of memory leaks. It eliminates some types of
problems and adds some others, but I think on the balance it should be better.
Reusing the user classes means we'd need to track changes to classpaths
efficiently, and this means we can do some interesting things on change:
- invalidate a task's outputs when the task implementation changes
- notify the tooling API client that the model may have changed.
And we can cache some interesting information derived from a classpath:
- the API for the classpath, which can be used for up-to-date checks
and compilation.
- plugin meta-data scanned from the classpath (eg
@Plugin(id='my-plugin') annotation, or plugin supplied services).
- test detection.
- the project model, or enough of the model to decide it the project's
outputs are up-to-date or not.
--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com
Join us at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA:
http://www.gradlesummit.com
