On 09/12/2011, at 1:52 AM, Szczepan Faber wrote: > Hey guys, > > Firstly, I'd like to recap the conversation a little bit so that we have a > shared understanding. > > 1. We don't want another environmental variable to control daemon parameters. > 2. (Quoting Adam) We want to have a config file in the project directory > somewhere (whatever that ends up being) that can declare the minimum heap and > permspace requirements of the build, and maybe other JVM args that are > required to launch the build. > > Secondly, I'd like to share my humble feedback on some of the things > mentioned on the thread :) > > It would be cool if we approach the problem incrementally. That is, first we > make it possible to configure the daemon vm parameters. Then, if we concur > that it is important at this point, we can discuss a different story, that is > what should be the default daemon vm args. IMHO, this discussion should > include: 1. consideration around backwards compatibility 2. compatibility in > the context of tooling api - slightly tricky due to the tooling api > compatibility contract and the way tooling api is used, e.g. under the hood > of the IDE plugins.
Here's the plan for milestone-7: There will be 3 properties available that affect how Gradle runs the build: * org.gradle.jvmargs - the JVM args required for the build. * org.gradle.daemon - whether the daemon is enabled or disabled by default. * org.gradle.daemon.idletimeout - for tweaking purposes. There will be a few places you can specify these, in increasing order of precedence: * In the 'gradle.properties' in the root directory of a build. This will allow you to enable the daemon by default for a particular build, or to declare the default JVM args to use for the build. * In '~/.gradle/gradle.properties'. This will allow the user to override these settings, or apply some settings globally. So, for example, on your CI machine, you can disable the daemon for all build, but still enable it for all your projects on a dev machine. * As a system property, via the -D command-line option. This will allow the user to override the above, or tweak for a particular invocation. * Using the --daemon/--no-daemon command-line options. Some things to note about this in milestone-7: This is very much experimental, so may change completely or (not very likely) disappear in milestone-8. There are a couple of restrictions: org.gradle.jvmargs will be honoured only by the tooling API and when you are running with the daemon. And org.gradle.daemon is ignored by the tooling API. It will always use the daemon. Also, we will still use the old default jvm args of -Xmx1024m -XX:MaxPermSize=256m if you don't specify anything. For milestone-8, the plan is: 1. Come up with a solution to allow org.gradle.jvmargs to be honoured when you're not using the daemon. 2. If this is all working ok for milestone-7, lock it down as an official interface. 3. Provide a way for the tooling API to override and/or configure these values. For (milestone-8 + 1): 1. Come up with some better defaults. > > Cheers! > > On Wed, Dec 7, 2011 at 7:57 PM, Kris De Volder <[email protected]> wrote: > > What kind of management does it need to do? i.e. what would very good > > management look like? > > I've already given some thoughts on this to Adam in the Gradle issue tracker > issue about 'runaway daemon'. > > But this is a hard question. I don't have the answer, just some random > thoughts :-) > > What I definitely don't want: > - multiple daemons parking themselves in memory practically forever. > > So one thing it needs to do is limit the number of daemons that are allowed > to exist concurrently. > I think the 'kill' logic should be more and more aggressive based on # of > daemons already in use. > And it would be nice if there was some API controlled way to set a hard limit > on the # of daemons allowed > to exist at the same time. > > The 'daemon manager' should be able to manage all Gradle daemons, across > gradle versions > (a common thing in my use has been multiple daemons get started because of > different gradle > versions). Sometimes one of those daemons is 'by accident' because gradle > version set wrong... > and only used once. But it sticks around for what seems forever. > > Also it has been a problem that daemons would get 'stuck' or something. And > then sit there > 'defunct' while another daemon got started in its place. This way, I've seen > 10+ concurrent daemons on my machine. > So daemons should have a robust 'self destruct'. If something is wrong, or if > they haven't been used in some time, they should just commit suicide. And > this mechanism should be robust and not itself fail or get stuck somehow. > > Another useful thing would be that I could somehow 'own my daemon' and kill > it when I know I am not > going to be using it again (e.g. if the IDE is shutdown, it's 'associated' > daemon probably should also > go away as it probably won't be used anymore). > > > > Another thing to consider: as you say, indeed the JVM never gives > > > memory back to the OS. The only way to really give it back is to > > > terminate the JVM (i.e. daemon). So I'd suggest not having the > > > daemon stick around for too long, unless it is really being used > > > quite frequently… > > > > At the moment, the default idle time is 3 hours. This will be user > > configurable though. > > I exaggerate, but 3 minutes would be better :-) > I see we are somewhat far apart on the meaning of 'frequently'. > > > > > > My compromise would be a combination of two strategies: > > > - be generous with max memory limits (so things work without much > > > manual tweaking) > > > - be conservative and 'kill' unused daemons of fairly quickly (so > > > unused resources are returned to OS) > > > > This sounds right to me, now we just need to agree on what the idle > > timeout should be :) > > Tricky :-). My feeling is... something that can be measured in minutes, not > hours :-) > > Maybe 10 minutes? Maybe 5 minutes, maybe 15 minutes (that's long!)... > 3 hours in my world... is practically equivalent to 'forever'. You might as > well have no timeout. > > Just put in to perspective: how much do you gain from not having to pay the > startup > cost once every 3 hours? Is that worth permanently sacrificing +1Gb of memory? > > If it is tweakable somehow via the tooling API it is less important what the > exact value would be, because I can tweak it from the IDE if I don't like > its default setting. > > > > > If we are too conservative we almost completely negate the user > > experience benefit of using the daemon. > > > > The same is true if you keep the daemon there for too long. Because it > degrades overall performance (memory given to the daemon does hurt > performance because it isn't available for other processes and OS level > caches). > > I'm more worried about overall performance degradation than speeding up the > occasional gradle command. > > This is especially the case *if* you reach the 'swap tipping point', where it > makes the difference between a usable or unusable system. Paying the startup > cost for Gradle degrades the experience, but doesn't really make my machine > unusable. > > This problem is worse for me, because as a Tooling API user, I do not have > the option of not using the daemon. A commandline user always has this option > if the daemon doesn't work for them. > > > > I believe, it just isn't worth the price to keep a daemon parked in > > > precious memory for extended periods of time, if it is not being > > > used. Rather than counting on paging it out, its much better to > > > shut it down. If it is paged out, it won't give you great > > > performance next time you try to use it anyway. > > > > It would be interesting to test if paging in is cheaper than > > bootstrapping Groovy or not. I'm not suggesting we do this. > > Testing a bit to find out doesn't hurt :-) > > Also keep in mind, I don't think the JVM is ever really 100% 'idle'. So even > when not doing anything, it is probably still touching some of the memory in > its heap every so often. So try to factor in a form of 'constant drag' on the > system just for having a parked daemon. > > > > > Shutting it down is costly though in terms of user experience so > > that's not without consequence. This is a fine balancing act. > > Keeping it there may be more costly in certain cases (if it causes paging). > Also... shutting it down, in itself doesn't really hurt user experience. > > It is the startup that hurts, not the shutdown. > > So the shutdown only hurts assuming that I'm actually going to be using > the same daemon again, soon. > > > No problem, no offence taken. I just had to clarify what I was > > saying. Your input is very valuable and I think we are increasing > > the understanding here. Although, personally I don't think I have a > > good idea on these defaults yet. > > That's good. Glad to hear that. I think it is good to keep an open mind about > this. And > also it should point to the importance of allowing the user to override > the defaults. You do your best to set sensible defaults, but accept that > they may not be right for every use case. > > I'm speaking as a Tooling API client here... and for me, there currently > is no way to tweak any of these defaults. > > Thanks for the interesting discussion :-) > > Kris > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > > > > > -- > Szczepan Faber > Principal engineer@gradleware > Lead@mockito -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
