> From: ja...@takari.io > Subject: Re: incremental compiler questions > Date: Sat, 10 Jan 2015 05:33:21 -0500 > To: dev@maven.apache.org > > > On Jan 10, 2015, at 3:16 AM, Mark Struberg <strub...@yahoo.de> wrote: > > > Hi Martin! > > > > The maven-compiler plugin already does this. But once a single change is > > detected then we need to recompile the _whole_ module because of the > > reasons explained by Igor. > > Pro of JDT: you can e.g. also see if there were only 'internal' changes and > > thus the other dependencies don't need to recompile. > > > > I'm 99.8, but not 100% convinced by that. Probably the other dependencies > > don't need any recompilation but maybe the whole internal behaviour changed > > and you still like to run your tests? Maybe we can simply ignore this and > > people should simply force a clean beforehand. > > > > In the project where we developed the incremental lifecycle forcing a clean > is not practical. It introduces a huge penalty when you're dealing with tens > of source generators and annotations processors with hundreds of modules. > Yes, it will take time to instrument all the plugins we care about but have > done so thus far and it is possible to reason about the incremental state of > the whole project as a consequence. Right now we are focused on individual > mojo executions but eventually we want global incremental state management > which is the path to full impact analysis. Reaching that state is when you > can reason about what classes and resources changing affect what tests and > you start running on the tests that are necessary. As Paul mentioned in the > hangout he's got 20 hours of tests. In our projects we have 90 minutes > functional tests on extremely powerful machines. But this is where the > biggest gains will be made but a lot of work needs to be done to get to that > point. > > > > > Basically al the incremental stuff boils down to the fact that the outcome > > is only determined by it's input. The problem is only to gather the right > > 'input'. E.g. your packaging relies not only on the compiled classes but > > also on resources and probably a manual include/exclude setting. Only the > > very plugin used to do the packaging will know. Thus EACH plugin in the > > whole chain needs to get adopted to incremental-awareness. Which is tons of > > work. > > > > It is not only determined by inputs but also configuration, and not only > changed inputs but deleted inputs in order to remove stale outputs: this is > incredibly important. The other critical difference in our implementation is > awareness of dependency relationships. To our knowledge we have the first, > complete and correct incremental mechanism for Maven. Simon says Buck has > something similar and I know of a couple commercial implementations. Our end > goal is full impact analysis and something more akin to a global state > machine. > > Yes, all the mojos you use must be instrumented correctly and consistently > and this is a huge amount of work. For our customer we started with the > default lifecycle, targeted plugins that were required to be instrumented and > we're just stepwise improving the support. We had a requirement for pure > speed in order to meet our wall time requirements but I believe the vast > majority of people doing enterprise builds care about 20-30 of the plugins > that exist and we'll eventually get coverage for those. > > > > > Another story is if the 'input' e.g. includes data from a database or you > > just like to re-run your unit tests for the sake of debugging. We need a > > switch to force some plugins to run even if the plugin itself cannot see a > > reason by simply looking at it's known 'input'. > > > > There's nothing intrinsically different in this case. It's a matter of > creating the correct graph and even if you had resources in databases or your > schema changes you need to know how to process a delta from said resource and > what is affected by those changes. There's nothing terribly magical here, the > hard part for us right now is what the API should look like (we're having a > lot of difficulty with that) and the sheer amount of work to instrument the > plugins. > > This was definitely an oversight when we started writing the system. We > didn't much think about incremental behaviour at all but it really is > necessary. > > For us the benefit is that once a plugin is instrumented with our incremental > API you get incremental behaviour on the CLI and inside of Eclipse and no > special integration needs to be written. Ultimately from one API we want > proper incremental behavior (and workspace resolution semantics) on the CLI, > the IDE and CI systems. This is a requirement on the path to correct and > efficient continuous delivery. It's all connected.
MG>Then Schema deltas *should* be able to update the configuration for incremental compilation? MG>Would a static configuration that listens for "schema deltas" for the incremental compiler next iteration be helpful? MG>knowing the graph at any point in time seems to be a key to understanding what collection requires re-compilation MG>can Kristians work be refactored to be able to produce graph snapshots? MG>as far as implementation I would suggest CLI first, CI (Jenkins?) second and IDE third..refactoring an IDE plugin that works in Eclipse, Idea and Netbeans from my experience is a very time consuming exercise. MG>Thanks for "thinking this through" Jason Mark and Igor MG>Thoughts? > > > > > LieGrue, > > strub > > > > > > > > > > > >> On Saturday, 10 January 2015, 2:04, Martin Gainty <mgai...@hotmail.com> > >> wrote: > >>> G ood Evening Igor- > >> > >> Instead of implementing eclipe jdt for incremental java compilation is > >> there: > >> > >> 1)any ability to refactor/reconfigure javac to read Last Modified Date of > >> source > >> file to enable incremental compilation? > >> 2)any concerns over generated class files..are they the same? > >> any deltas ..if so ...what are they? > >> 3)can we implement a static settings.xml or other static configuration > >> file that > >> manfred suggested for eclipse jdt? > >> 4)any ability to switch compiler from javac to jdt as an attribute in > >> maven-compiler-plugin? > >> > >> Thanks for a great presentation on Maven Dev hangout > >> Martin > >> ______________________________________________ > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > > For additional commands, e-mail: dev-h...@maven.apache.org > > > > Thanks, > > Jason > > ---------------------------------------------------------- > Jason van Zyl > Founder, Apache Maven > http://twitter.com/jvanzyl > http://twitter.com/takari_io > --------------------------------------------------------- > > believe nothing, no matter where you read it, > or who has said it, > not even if i have said it, > unless it agrees with your own reason > and your own common sense. > > -- Buddha > > > > > > > > >
