Szcepan, thanks for bringing things back down to earth. 1. We don't want another environmental variable to control daemon parameters. Agreed.
2. (Quoting Adam) We want to have a config file in the project directory somewhere (whatever that ends up being) that can declare the minimum heap and permspace requirements of the build, and maybe other JVM args that are required to launch the build. Right, that sounds very good to me. Especially if the file format can be considered 'public'. This would allow the IDE to read and modify it (and provide simple tools for editing / viewing those parameters. If the format is 'public' we may not need API to set those params. (I.e. the file format becomes part of the API spec). One small observation, Daemon timeout value seems to be different kind of param than say 'heap requirements'. It feels to me like timeout value would be something connected to overall daemon management, i.e. it isn't a project-specific requirement. We can drop discussions surrounding daemon management for now. (Not that I don't think they are important... but further discussion won't add much useful info now and Adam is probably already well enough aware of my concerns in this area :-) Kris ----- Original Message ----- > Hey guys, > Firstly, I'd like to recap the conversation a little bit so that we > have a shared understanding. > 1. We don't want another environmental variable to control daemon > parameters. > 2. (Quoting Adam) We want to have a config file in the project > directory somewhere (whatever that ends up being) that can declare > the minimum heap and permspace requirements of the build, and maybe > other JVM args that are required to launch the build. > Secondly, I'd like to share my humble feedback on some of the things > mentioned on the thread :) > It would be cool if we approach the problem incrementally. That is, > first we make it possible to configure the daemon vm parameters. > Then, if we concur that it is important at this point, we can > discuss a different story, that is what should be the default daemon > vm args. IMHO, this discussion should include: 1. consideration > around backwards compatibility 2. compatibility in the context of > tooling api - slightly tricky due to the tooling api compatibility > contract and the way tooling api is used, e.g. under the hood of the > IDE plugins. > Cheers! > On Wed, Dec 7, 2011 at 7:57 PM, Kris De Volder < [email protected] > > wrote: > > > What kind of management does it need to do? i.e. what would very > > > good management look like? > > > I've already given some thoughts on this to Adam in the Gradle > > issue > > tracker issue about 'runaway daemon'. > > > But this is a hard question. I don't have the answer, just some > > random thoughts :-) > > > What I definitely don't want: > > > - multiple daemons parking themselves in memory practically > > forever. > > > So one thing it needs to do is limit the number of daemons that are > > allowed to exist concurrently. > > > I think the 'kill' logic should be more and more aggressive based > > on > > # of daemons already in use. > > > And it would be nice if there was some API controlled way to set a > > hard limit on the # of daemons allowed > > > to exist at the same time. > > > The 'daemon manager' should be able to manage all Gradle daemons, > > across gradle versions > > > (a common thing in my use has been multiple daemons get started > > because of different gradle > > > versions). Sometimes one of those daemons is 'by accident' because > > gradle version set wrong... > > > and only used once. But it sticks around for what seems forever. > > > Also it has been a problem that daemons would get 'stuck' or > > something. And then sit there > > > 'defunct' while another daemon got started in its place. This way, > > I've seen 10+ concurrent daemons on my machine. > > > So daemons should have a robust 'self destruct'. If something is > > wrong, or if they haven't been used in some time, they should just > > commit suicide. And this mechanism should be robust and not itself > > fail or get stuck somehow. > > > Another useful thing would be that I could somehow 'own my daemon' > > and kill it when I know I am not > > > going to be using it again (e.g. if the IDE is shutdown, it's > > 'associated' daemon probably should also > > > go away as it probably won't be used anymore). > > > > > Another thing to consider: as you say, indeed the JVM never > > > > gives > > > > > memory back to the OS. The only way to really give it back is > > > > to > > > > > terminate the JVM (i.e. daemon). So I'd suggest not having the > > > > > daemon stick around for too long, unless it is really being > > > > used > > > > > quite frequently… > > > > > > > > At the moment, the default idle time is 3 hours. This will be > > > user > > > > configurable though. > > > I exaggerate, but 3 minutes would be better :-) > > > I see we are somewhat far apart on the meaning of 'frequently'. > > > > > > > > > My compromise would be a combination of two strategies: > > > > > - be generous with max memory limits (so things work without > > > > much > > > > > manual tweaking) > > > > > - be conservative and 'kill' unused daemons of fairly quickly > > > > (so > > > > > unused resources are returned to OS) > > > > > > > > This sounds right to me, now we just need to agree on what the > > > idle > > > > timeout should be :) > > > Tricky :-). My feeling is... something that can be measured in > > minutes, not hours :-) > > > Maybe 10 minutes? Maybe 5 minutes, maybe 15 minutes (that's > > long!)... > > > 3 hours in my world... is practically equivalent to 'forever'. You > > might as well have no timeout. > > > Just put in to perspective: how much do you gain from not having to > > pay the startup > > > cost once every 3 hours? Is that worth permanently sacrificing +1Gb > > of memory? > > > If it is tweakable somehow via the tooling API it is less important > > what the > > > exact value would be, because I can tweak it from the IDE if I > > don't > > like > > > its default setting. > > > > > > > > If we are too conservative we almost completely negate the user > > > > experience benefit of using the daemon. > > > > > > > The same is true if you keep the daemon there for too long. Because > > it degrades overall performance (memory given to the daemon does > > hurt performance because it isn't available for other processes and > > OS level caches). > > > I'm more worried about overall performance degradation than > > speeding > > up the occasional gradle command. > > > This is especially the case *if* you reach the 'swap tipping > > point', > > where it makes the difference between a usable or unusable system. > > Paying the startup cost for Gradle degrades the experience, but > > doesn't really make my machine unusable. > > > This problem is worse for me, because as a Tooling API user, I do > > not > > have the option of not using the daemon. A commandline user always > > has this option if the daemon doesn't work for them. > > > > > I believe, it just isn't worth the price to keep a daemon > > > > parked > > > > in > > > > > precious memory for extended periods of time, if it is not > > > > being > > > > > used. Rather than counting on paging it out, its much better to > > > > > shut it down. If it is paged out, it won't give you great > > > > > performance next time you try to use it anyway. > > > > > > > > It would be interesting to test if paging in is cheaper than > > > > bootstrapping Groovy or not. I'm not suggesting we do this. > > > Testing a bit to find out doesn't hurt :-) > > > Also keep in mind, I don't think the JVM is ever really 100% > > 'idle'. > > So even when not doing anything, it is probably still touching some > > of the memory in its heap every so often. So try to factor in a > > form > > of 'constant drag' on the system just for having a parked daemon. > > > > > > > > Shutting it down is costly though in terms of user experience so > > > > that's not without consequence. This is a fine balancing act. > > > Keeping it there may be more costly in certain cases (if it causes > > paging). > > > Also... shutting it down, in itself doesn't really hurt user > > experience. > > > It is the startup that hurts, not the shutdown. > > > So the shutdown only hurts assuming that I'm actually going to be > > using > > > the same daemon again, soon. > > > > No problem, no offence taken. I just had to clarify what I was > > > > saying. Your input is very valuable and I think we are increasing > > > > the understanding here. Although, personally I don't think I have > > > a > > > > good idea on these defaults yet. > > > That's good. Glad to hear that. I think it is good to keep an open > > mind about this. And > > > also it should point to the importance of allowing the user to > > override > > > the defaults. You do your best to set sensible defaults, but accept > > that > > > they may not be right for every use case. > > > I'm speaking as a Tooling API client here... and for me, there > > currently > > > is no way to tweak any of these defaults. > > > Thanks for the interesting discussion :-) > > > Kris > > > --------------------------------------------------------------------- > > > To unsubscribe from this list, please visit: > > > http://xircles.codehaus.org/manage_email > > -- > Szczepan Faber > Principal engineer@gradleware > Lead@mockito
