RE: incremental compiler questions
From: ja...@takari.io Subject: Re: incremental compiler questions Date: Sat, 10 Jan 2015 05:33:21 -0500 To: dev@maven.apache.org On Jan 10, 2015, at 3:16 AM, Mark Struberg strub...@yahoo.de wrote: Hi Martin! The maven-compiler plugin already does this. But once a single change is detected then we need to recompile the _whole_ module because of the reasons explained by Igor. Pro of JDT: you can e.g. also see if there were only 'internal' changes and thus the other dependencies don't need to recompile. I'm 99.8, but not 100% convinced by that. Probably the other dependencies don't need any recompilation but maybe the whole internal behaviour changed and you still like to run your tests? Maybe we can simply ignore this and people should simply force a clean beforehand. In the project where we developed the incremental lifecycle forcing a clean is not practical. It introduces a huge penalty when you're dealing with tens of source generators and annotations processors with hundreds of modules. Yes, it will take time to instrument all the plugins we care about but have done so thus far and it is possible to reason about the incremental state of the whole project as a consequence. Right now we are focused on individual mojo executions but eventually we want global incremental state management which is the path to full impact analysis. Reaching that state is when you can reason about what classes and resources changing affect what tests and you start running on the tests that are necessary. As Paul mentioned in the hangout he's got 20 hours of tests. In our projects we have 90 minutes functional tests on extremely powerful machines. But this is where the biggest gains will be made but a lot of work needs to be done to get to that point. Basically al the incremental stuff boils down to the fact that the outcome is only determined by it's input. The problem is only to gather the right 'input'. E.g. your packaging relies not only on the compiled classes but also on resources and probably a manual include/exclude setting. Only the very plugin used to do the packaging will know. Thus EACH plugin in the whole chain needs to get adopted to incremental-awareness. Which is tons of work. It is not only determined by inputs but also configuration, and not only changed inputs but deleted inputs in order to remove stale outputs: this is incredibly important. The other critical difference in our implementation is awareness of dependency relationships. To our knowledge we have the first, complete and correct incremental mechanism for Maven. Simon says Buck has something similar and I know of a couple commercial implementations. Our end goal is full impact analysis and something more akin to a global state machine. Yes, all the mojos you use must be instrumented correctly and consistently and this is a huge amount of work. For our customer we started with the default lifecycle, targeted plugins that were required to be instrumented and we're just stepwise improving the support. We had a requirement for pure speed in order to meet our wall time requirements but I believe the vast majority of people doing enterprise builds care about 20-30 of the plugins that exist and we'll eventually get coverage for those. Another story is if the 'input' e.g. includes data from a database or you just like to re-run your unit tests for the sake of debugging. We need a switch to force some plugins to run even if the plugin itself cannot see a reason by simply looking at it's known 'input'. There's nothing intrinsically different in this case. It's a matter of creating the correct graph and even if you had resources in databases or your schema changes you need to know how to process a delta from said resource and what is affected by those changes. There's nothing terribly magical here, the hard part for us right now is what the API should look like (we're having a lot of difficulty with that) and the sheer amount of work to instrument the plugins. This was definitely an oversight when we started writing the system. We didn't much think about incremental behaviour at all but it really is necessary. For us the benefit is that once a plugin is instrumented with our incremental API you get incremental behaviour on the CLI and inside of Eclipse and no special integration needs to be written. Ultimately from one API we want proper incremental behavior (and workspace resolution semantics) on the CLI, the IDE and CI systems. This is a requirement on the path to correct and efficient continuous delivery. It's all connected. MGThen Schema deltas *should* be able to update the configuration for incremental compilation? MGWould a static configuration that listens for schema deltas for the incremental compiler next iteration be helpful? MGknowing the graph at any point
Re: incremental compiler questions
Hi Martin! The maven-compiler plugin already does this. But once a single change is detected then we need to recompile the _whole_ module because of the reasons explained by Igor. Pro of JDT: you can e.g. also see if there were only 'internal' changes and thus the other dependencies don't need to recompile. I'm 99.8, but not 100% convinced by that. Probably the other dependencies don't need any recompilation but maybe the whole internal behaviour changed and you still like to run your tests? Maybe we can simply ignore this and people should simply force a clean beforehand. Basically al the incremental stuff boils down to the fact that the outcome is only determined by it's input. The problem is only to gather the right 'input'. E.g. your packaging relies not only on the compiled classes but also on resources and probably a manual include/exclude setting. Only the very plugin used to do the packaging will know. Thus EACH plugin in the whole chain needs to get adopted to incremental-awareness. Which is tons of work. Another story is if the 'input' e.g. includes data from a database or you just like to re-run your unit tests for the sake of debugging. We need a switch to force some plugins to run even if the plugin itself cannot see a reason by simply looking at it's known 'input'. LieGrue, strub On Saturday, 10 January 2015, 2:04, Martin Gainty mgai...@hotmail.com wrote: G ood Evening Igor- Instead of implementing eclipe jdt for incremental java compilation is there: 1)any ability to refactor/reconfigure javac to read Last Modified Date of source file to enable incremental compilation? 2)any concerns over generated class files..are they the same? any deltas ..if so ...what are they? 3)can we implement a static settings.xml or other static configuration file that manfred suggested for eclipse jdt? 4)any ability to switch compiler from javac to jdt as an attribute in maven-compiler-plugin? Thanks for a great presentation on Maven Dev hangout Martin __ - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: incremental compiler questions
On Jan 10, 2015, at 3:16 AM, Mark Struberg strub...@yahoo.de wrote: Hi Martin! The maven-compiler plugin already does this. But once a single change is detected then we need to recompile the _whole_ module because of the reasons explained by Igor. Pro of JDT: you can e.g. also see if there were only 'internal' changes and thus the other dependencies don't need to recompile. I'm 99.8, but not 100% convinced by that. Probably the other dependencies don't need any recompilation but maybe the whole internal behaviour changed and you still like to run your tests? Maybe we can simply ignore this and people should simply force a clean beforehand. In the project where we developed the incremental lifecycle forcing a clean is not practical. It introduces a huge penalty when you're dealing with tens of source generators and annotations processors with hundreds of modules. Yes, it will take time to instrument all the plugins we care about but have done so thus far and it is possible to reason about the incremental state of the whole project as a consequence. Right now we are focused on individual mojo executions but eventually we want global incremental state management which is the path to full impact analysis. Reaching that state is when you can reason about what classes and resources changing affect what tests and you start running on the tests that are necessary. As Paul mentioned in the hangout he's got 20 hours of tests. In our projects we have 90 minutes functional tests on extremely powerful machines. But this is where the biggest gains will be made but a lot of work needs to be done to get to that point. Basically al the incremental stuff boils down to the fact that the outcome is only determined by it's input. The problem is only to gather the right 'input'. E.g. your packaging relies not only on the compiled classes but also on resources and probably a manual include/exclude setting. Only the very plugin used to do the packaging will know. Thus EACH plugin in the whole chain needs to get adopted to incremental-awareness. Which is tons of work. It is not only determined by inputs but also configuration, and not only changed inputs but deleted inputs in order to remove stale outputs: this is incredibly important. The other critical difference in our implementation is awareness of dependency relationships. To our knowledge we have the first, complete and correct incremental mechanism for Maven. Simon says Buck has something similar and I know of a couple commercial implementations. Our end goal is full impact analysis and something more akin to a global state machine. Yes, all the mojos you use must be instrumented correctly and consistently and this is a huge amount of work. For our customer we started with the default lifecycle, targeted plugins that were required to be instrumented and we're just stepwise improving the support. We had a requirement for pure speed in order to meet our wall time requirements but I believe the vast majority of people doing enterprise builds care about 20-30 of the plugins that exist and we'll eventually get coverage for those. Another story is if the 'input' e.g. includes data from a database or you just like to re-run your unit tests for the sake of debugging. We need a switch to force some plugins to run even if the plugin itself cannot see a reason by simply looking at it's known 'input'. There's nothing intrinsically different in this case. It's a matter of creating the correct graph and even if you had resources in databases or your schema changes you need to know how to process a delta from said resource and what is affected by those changes. There's nothing terribly magical here, the hard part for us right now is what the API should look like (we're having a lot of difficulty with that) and the sheer amount of work to instrument the plugins. This was definitely an oversight when we started writing the system. We didn't much think about incremental behaviour at all but it really is necessary. For us the benefit is that once a plugin is instrumented with our incremental API you get incremental behaviour on the CLI and inside of Eclipse and no special integration needs to be written. Ultimately from one API we want proper incremental behavior (and workspace resolution semantics) on the CLI, the IDE and CI systems. This is a requirement on the path to correct and efficient continuous delivery. It's all connected. LieGrue, strub On Saturday, 10 January 2015, 2:04, Martin Gainty mgai...@hotmail.com wrote: G ood Evening Igor- Instead of implementing eclipe jdt for incremental java compilation is there: 1)any ability to refactor/reconfigure javac to read Last Modified Date of source file to enable incremental compilation? 2)any concerns over generated class files..are they the same? any deltas ..if so ...what are they? 3)can we implement a static
incremental compiler questions
Good Evening Igor- Instead of implementing eclipe jdt for incremental java compilation is there: 1)any ability to refactor/reconfigure javac to read Last Modified Date of source file to enable incremental compilation? 2)any concerns over generated class files..are they the same? any deltas ..if so ...what are they? 3)can we implement a static settings.xml or other static configuration file that manfred suggested for eclipse jdt? 4)any ability to switch compiler from javac to jdt as an attribute in maven-compiler-plugin? Thanks for a great presentation on Maven Dev hangout Martin __
Re: incremental compiler questions
I am not sure I understand your questions, maybe you can explain you what you are trying to achieve and I can tell how to do it. Few general comments As I tried to explain in the hangout, proper incremental compilation requires type reference information. Consider two classes, A and B extends A, and C. When A.java changes, it is not enough to recompile just this source, it is also necessary to recompile B.java. Javac does not provide type reference information, so the only safe way to handle any change is to recompile everything. Eclipse JDT compiler is production ready and it produces spec-compliant class files. I believe all eclipse ecosystem is built with this compiler and I've seen many projects outside of eclipse use it too. Eclipse jdt is a name of Eclipse project that produces Java Development Tools (hence the name). Spec-compliant java compiler is one of deliverables of this project. -- Regards, Igor On 2015-01-09 20:02, Martin Gainty wrote: Good Evening Igor- Instead of implementing eclipe jdt for incremental java compilation is there: 1)any ability to refactor/reconfigure javac to read Last Modified Date of source file to enable incremental compilation? 2)any concerns over generated class files..are they the same? any deltas ..if so ...what are they? 3)can we implement a static settings.xml or other static configuration file that manfred suggested for eclipse jdt? 4)any ability to switch compiler from javac to jdt as an attribute in maven-compiler-plugin? Thanks for a great presentation on Maven Dev hangout Martin __ - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
