RE: incremental compiler questions
> From: ja...@takari.io > Subject: Re: incremental compiler questions > Date: Sat, 10 Jan 2015 05:33:21 -0500 > To: dev@maven.apache.org > > > On Jan 10, 2015, at 3:16 AM, Mark Struberg wrote: > > > Hi Martin! > > > > The maven-compiler plugin already does this. But once a single change is > > detected then we need to recompile the _whole_ module because of the > > reasons explained by Igor. > > Pro of JDT: you can e.g. also see if there were only 'internal' changes and > > thus the other dependencies don't need to recompile. > > > > I'm 99.8, but not 100% convinced by that. Probably the other dependencies > > don't need any recompilation but maybe the whole internal behaviour changed > > and you still like to run your tests? Maybe we can simply ignore this and > > people should simply force a clean beforehand. > > > > In the project where we developed the incremental lifecycle forcing a clean > is not practical. It introduces a huge penalty when you're dealing with tens > of source generators and annotations processors with hundreds of modules. > Yes, it will take time to instrument all the plugins we care about but have > done so thus far and it is possible to reason about the incremental state of > the whole project as a consequence. Right now we are focused on individual > mojo executions but eventually we want global incremental state management > which is the path to full impact analysis. Reaching that state is when you > can reason about what classes and resources changing affect what tests and > you start running on the tests that are necessary. As Paul mentioned in the > hangout he's got 20 hours of tests. In our projects we have 90 minutes > functional tests on extremely powerful machines. But this is where the > biggest gains will be made but a lot of work needs to be done to get to that > point. > > > > > Basically al the incremental stuff boils down to the fact that the outcome > > is only determined by it's input. The problem is only to gather the right > > 'input'. E.g. your packaging relies not only on the compiled classes but > > also on resources and probably a manual include/exclude setting. Only the > > very plugin used to do the packaging will know. Thus EACH plugin in the > > whole chain needs to get adopted to incremental-awareness. Which is tons of > > work. > > > > It is not only determined by inputs but also configuration, and not only > changed inputs but deleted inputs in order to remove stale outputs: this is > incredibly important. The other critical difference in our implementation is > awareness of dependency relationships. To our knowledge we have the first, > complete and correct incremental mechanism for Maven. Simon says Buck has > something similar and I know of a couple commercial implementations. Our end > goal is full impact analysis and something more akin to a global state > machine. > > Yes, all the mojos you use must be instrumented correctly and consistently > and this is a huge amount of work. For our customer we started with the > default lifecycle, targeted plugins that were required to be instrumented and > we're just stepwise improving the support. We had a requirement for pure > speed in order to meet our wall time requirements but I believe the vast > majority of people doing enterprise builds care about 20-30 of the plugins > that exist and we'll eventually get coverage for those. > > > > > Another story is if the 'input' e.g. includes data from a database or you > > just like to re-run your unit tests for the sake of debugging. We need a > > switch to force some plugins to run even if the plugin itself cannot see a > > reason by simply looking at it's known 'input'. > > > > There's nothing intrinsically different in this case. It's a matter of > creating the correct graph and even if you had resources in databases or your > schema changes you need to know how to process a delta from said resource and > what is affected by those changes. There's nothing terribly magical here, the > hard part for us right now is what the API should look like (we're having a > lot of difficulty with that) and the sheer amount of work to instrument the > plugins. > > This was definitely an oversight when we started writing the system. We > didn't much think about incremental behaviour at all but it really is > necessary. > > For us the benefit is that once a plugin is instrumented with our incremental > API you get incremental behaviour on the CLI and inside of Eclipse and no > special integration needs to be written. Ultimately from one API we want > proper incremental behavior (and workspace resolution semantics) on the CLI, > the IDE and CI systems. This is a requirement on the path to correct and > efficient continuous delivery. It's all connected. MG>Then Schema deltas *should* be able to update the configuration for incremental compilation? MG>Would a static configuration that listens for "s
Re: Preview release: Plexus Archiver multithreaded Zip edition
guys, if there are any arguments, I can give you Word document with the jar and test instructions. I was testing this with one Russian guys on Unix and I was testing on Windows. We presented this in a discussion at LinkedIn. The NIO work nicely with big chunks you are about to write; But if you want to write just one integer to IO, then NIO is just wrong path. Simply make writes into 256KB buffer and then write it down when buffer overflows. You will see differences between native and heap buffer. The typical heap ByteBuffer uses arrays which needs two machine cycles per read or write. Reason is that one operation is array range check and then read/write. The native ByteBuffer is a bit risky, but 2-times faster than heap buffer. Remember that MappedByteBuffer mapped file system with memory, and JVM reserved 256MB for native byte buffers. Using DMA the MappedByteBuffer.write(256KB ByteBuffer) es very fast not because of file system, but because of DMA memory access. So what i did, I took file (600MB and 1GB) one and made a copy into another file. Entire application consumed 8MB (4MB just for JVM) and performed at 400MB/s. I made a test experiment and used different size of Buffers: 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192 KB and the minimum latency on Windows was with 256KB and most fast. The measurement was very simple. I used ESET which is bind in kernel, and "Watch activity" tool gave me this measurement results. The same test was done on Unix & Sun machine with RAID and the minimum latency was not such significant - even worse than with Windows. -- View this message in context: http://maven.40175.n5.nabble.com/Preview-release-Plexus-Archiver-multithreaded-Zip-edition-tp5822942p5823154.html Sent from the Maven Developers mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: [VOTE] Release Apache Maven WAR Plugin version 2.6
Here is my +1 On 1/8/15 9:21 PM, Karl Heinz Marbaise wrote: Hi, We solved 12 issues: http://jira.codehaus.org/secure/ReleaseNote.jspa?projectId=11150&version=20681 There are still a couple of issues left in JIRA: http://jira.codehaus.org/issues/?jql=project%20%3D%20MWAR%20AND%20status%20%3D%20Open%20ORDER%20BY%20key%20DESC%2C%20priority%20DESC Staging repo: https://repository.apache.org/content/repositories/maven-1117 https://repository.apache.org/content/repositories/maven-1117/org/apache/maven/plugins/maven-war-plugin/2.6/maven-war-plugin-2.6-source-release.zip Source release checksum(s): maven-war-plugin-2.6-source-release.zip sha1: d687ff646f89253e79a7fa7e439ae57acd8f893 Staging site: http://maven.apache.org/plugins-archives/maven-war-plugin-LATEST/ Guide to testing staged releases: http://maven.apache.org/guides/development/guide-testing-releases.html Vote open for 72 hours. [ ] +1 [ ] +0 [ ] -1 Kind regards Karl Heinz Marbaise - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org Mit freundlichem Gruß Karl-Heinz Marbaise -- SoftwareEntwicklung Beratung SchulungTel.: +49 (0) 2405 / 415 893 Dipl.Ing.(FH) Karl-Heinz MarbaiseICQ#: 135949029 Hauptstrasse 177 USt.IdNr: DE191347579 52146 Würselen http://www.soebes.de - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
[RESULT] [VOTE] Release Apache Maven WAR Plugin version 2.6
Hi, The vote has passed with the following result: +1 (binding): Jason van Zyl, Mark Struberg, Karl Heinz Marbaise +1 (non binding): none. I will promote the artifacts to the central repo. Kind regards Karl Heinz Marbaise - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: maven git commit: better plugin/extensions realm parent classloader
please:
1. create a Jira issue
2. update our Class loading documentation [1] with MNG and Maven version
reference
then ideally, having a core IT to show what difference is expected from new
code would be great (I don't know how hard this one can be)
Regards,
Hervé
[1] https://cwiki.apache.org/confluence/display/MAVEN/Maven+3.x+Class+Loading
Le jeudi 8 janvier 2015 13:09:10 ifedore...@apache.org a écrit :
> Repository: maven
> Updated Branches:
> refs/heads/master 5f71f9789 -> bb4988496
>
>
> better plugin/extensions realm parent classloader
>
> Signed-off-by: Igor Fedorenko
>
>
> Project: http://git-wip-us.apache.org/repos/asf/maven/repo
> Commit: http://git-wip-us.apache.org/repos/asf/maven/commit/bb498849
> Tree: http://git-wip-us.apache.org/repos/asf/maven/tree/bb498849
> Diff: http://git-wip-us.apache.org/repos/asf/maven/diff/bb498849
>
> Branch: refs/heads/master
> Commit: bb4988496a0e3b50ee5a1922bcd54f731eb2d5b2
> Parents: 5f71f97
> Author: Igor Fedorenko
> Authored: Thu Jan 8 08:08:42 2015 -0500
> Committer: Igor Fedorenko
> Committed: Thu Jan 8 08:08:47 2015 -0500
>
> --
> .../maven/classrealm/DefaultClassRealmManager.java| 14 --
> 1 file changed, 12 insertions(+), 2 deletions(-)
> --
>
>
> http://git-wip-us.apache.org/repos/asf/maven/blob/bb498849/maven-core/src/ma
> in/java/org/apache/maven/classrealm/DefaultClassRealmManager.java
> -- diff
> --git
> a/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmMan
> ager.java
> b/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmMan
> ager.java index 3584d53..a5bb352 100644
> ---
> a/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmMan
> ager.java +++
> b/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmMan
> ager.java @@ -59,6 +59,16 @@ public class DefaultClassRealmManager
> implements ClassRealmManager
> {
>
> +/**
> + * During normal command line build, ClassWorld is loaded by jvm system
> classloader, which only includes + * plexus-classworlds jar and
> possibly javaagent classes, see http://jira.codehaus.org/browse/MNG-4747. +
> *
> + * Using ClassWorld to determine plugin/extensions realm parent
> classloaders gives m2e and integration test harness + * flexibility to
> load multiple version of maven into dedicated classloaders without assuming
> state of jvm system + * classloader.
> + */
> +private static final ClassLoader PARENT_CLASSLOADER =
> ClassWorld.class.getClassLoader(); +
> @Requirement
> private Logger logger;
>
> @@ -329,7 +339,7 @@ public class DefaultClassRealmManager
> throw new IllegalArgumentException( "extension plugin missing"
> ); }
>
> -ClassLoader parent = ClassLoader.getSystemClassLoader();
> +ClassLoader parent = PARENT_CLASSLOADER;
>
> Map foreignImports =
> Collections.singletonMap( "",
> getMavenApiRealm() ); @@ -347,7 +357,7 @@ public class
> DefaultClassRealmManager
>
> if ( parent == null )
> {
> -parent = ClassLoader.getSystemClassLoader();
> +parent = PARENT_CLASSLOADER;
> }
>
> return createRealm( getKey( plugin, false ), RealmType.Plugin,
> parent, parentImports, foreignImports, artifacts );
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: maven git commit: better plugin/extensions realm parent classloader
Opened https://jira.codehaus.org/browse/MNG-5752, thank you for the
reminder.
The change does not affect classloading of "normal" maven runtime, so
there is no need to update the diagram or documentation. Speaking of
which, that documentation is somewhat outdated and does not mention core
extensions realm, for example. I meant to update it some time ago, but
could not find the sources of the diagram. Does anyone know where they are?
I do not believe this change deserves dedicated integration test, at
least not in maven. There is only so many ways you can access system
classloader and if anyone decides to change this in the future javadoc
comment should give enough context.
--
Regards,
Igor
On 2015-01-11 11:17, Hervé BOUTEMY wrote:
please:
1. create a Jira issue
2. update our Class loading documentation [1] with MNG and Maven version
reference
then ideally, having a core IT to show what difference is expected from new
code would be great (I don't know how hard this one can be)
Regards,
Hervé
[1] https://cwiki.apache.org/confluence/display/MAVEN/Maven+3.x+Class+Loading
Le jeudi 8 janvier 2015 13:09:10 ifedore...@apache.org a écrit :
Repository: maven
Updated Branches:
refs/heads/master 5f71f9789 -> bb4988496
better plugin/extensions realm parent classloader
Signed-off-by: Igor Fedorenko
Project: http://git-wip-us.apache.org/repos/asf/maven/repo
Commit: http://git-wip-us.apache.org/repos/asf/maven/commit/bb498849
Tree: http://git-wip-us.apache.org/repos/asf/maven/tree/bb498849
Diff: http://git-wip-us.apache.org/repos/asf/maven/diff/bb498849
Branch: refs/heads/master
Commit: bb4988496a0e3b50ee5a1922bcd54f731eb2d5b2
Parents: 5f71f97
Author: Igor Fedorenko
Authored: Thu Jan 8 08:08:42 2015 -0500
Committer: Igor Fedorenko
Committed: Thu Jan 8 08:08:47 2015 -0500
--
.../maven/classrealm/DefaultClassRealmManager.java| 14 --
1 file changed, 12 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/maven/blob/bb498849/maven-core/src/ma
in/java/org/apache/maven/classrealm/DefaultClassRealmManager.java
-- diff
--git
a/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmMan
ager.java
b/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmMan
ager.java index 3584d53..a5bb352 100644
---
a/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmMan
ager.java +++
b/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmMan
ager.java @@ -59,6 +59,16 @@ public class DefaultClassRealmManager
implements ClassRealmManager
{
+/**
+ * During normal command line build, ClassWorld is loaded by jvm system
classloader, which only includes + * plexus-classworlds jar and
possibly javaagent classes, see http://jira.codehaus.org/browse/MNG-4747. +
*
+ * Using ClassWorld to determine plugin/extensions realm parent
classloaders gives m2e and integration test harness + * flexibility to
load multiple version of maven into dedicated classloaders without assuming
state of jvm system + * classloader.
+ */
+private static final ClassLoader PARENT_CLASSLOADER =
ClassWorld.class.getClassLoader(); +
@Requirement
private Logger logger;
@@ -329,7 +339,7 @@ public class DefaultClassRealmManager
throw new IllegalArgumentException( "extension plugin missing"
); }
-ClassLoader parent = ClassLoader.getSystemClassLoader();
+ClassLoader parent = PARENT_CLASSLOADER;
Map foreignImports =
Collections.singletonMap( "",
getMavenApiRealm() ); @@ -347,7 +357,7 @@ public class
DefaultClassRealmManager
if ( parent == null )
{
-parent = ClassLoader.getSystemClassLoader();
+parent = PARENT_CLASSLOADER;
}
return createRealm( getKey( plugin, false ), RealmType.Plugin,
parent, parentImports, foreignImports, artifacts );
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: maven git commit: better plugin/extensions realm parent classloader
Le dimanche 11 janvier 2015 12:19:01 Igor Fedorenko a écrit :
> Opened https://jira.codehaus.org/browse/MNG-5752, thank you for the
> reminder.
thank you: I now better understand the reasoning
>
> The change does not affect classloading of "normal" maven runtime, so
> there is no need to update the diagram or documentation.
IIUC, in the Wiki, what is currently described as "system classloader" is now
"initial classloader, containing classworlds", which is the system when using
CLI but may be another classloader in other situations where Maven is
integrated
> Speaking of
> which, that documentation is somewhat outdated and does not mention core
> extensions realm, for example. I meant to update it some time ago, but
> could not find the sources of the diagram. Does anyone know where they are?
if you look at attachements, there is a svg associated with the png image
>
> I do not believe this change deserves dedicated integration test, at
> least not in maven. There is only so many ways you can access system
> classloader and if anyone decides to change this in the future javadoc
> comment should give enough context.
ok
Regards,
Hervé
>
> --
> Regards,
> Igor
>
> On 2015-01-11 11:17, Hervé BOUTEMY wrote:
> > please:
> > 1. create a Jira issue
> > 2. update our Class loading documentation [1] with MNG and Maven version
> > reference
> >
> > then ideally, having a core IT to show what difference is expected from
> > new
> > code would be great (I don't know how hard this one can be)
> >
> > Regards,
> >
> > Hervé
> >
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/MAVEN/Maven+3.x+Class+Loading>
> > Le jeudi 8 janvier 2015 13:09:10 ifedore...@apache.org a écrit :
> >> Repository: maven
> >>
> >> Updated Branches:
> >>refs/heads/master 5f71f9789 -> bb4988496
> >>
> >> better plugin/extensions realm parent classloader
> >>
> >> Signed-off-by: Igor Fedorenko
> >>
> >>
> >> Project: http://git-wip-us.apache.org/repos/asf/maven/repo
> >> Commit: http://git-wip-us.apache.org/repos/asf/maven/commit/bb498849
> >> Tree: http://git-wip-us.apache.org/repos/asf/maven/tree/bb498849
> >> Diff: http://git-wip-us.apache.org/repos/asf/maven/diff/bb498849
> >>
> >> Branch: refs/heads/master
> >> Commit: bb4988496a0e3b50ee5a1922bcd54f731eb2d5b2
> >> Parents: 5f71f97
> >> Author: Igor Fedorenko
> >> Authored: Thu Jan 8 08:08:42 2015 -0500
> >> Committer: Igor Fedorenko
> >> Committed: Thu Jan 8 08:08:47 2015 -0500
> >>
> >> --
> >>
> >> .../maven/classrealm/DefaultClassRealmManager.java| 14
> >> --
> >> 1 file changed, 12 insertions(+), 2 deletions(-)
> >>
> >> --
> >>
> >>
> >> http://git-wip-us.apache.org/repos/asf/maven/blob/bb498849/maven-core/src
> >> /ma in/java/org/apache/maven/classrealm/DefaultClassRealmManager.java
> >> --
> >> diff
> >> --git
> >> a/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmM
> >> an
> >> ager.java
> >> b/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmM
> >> an
> >> ager.java index 3584d53..a5bb352 100644
> >> ---
> >> a/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmM
> >> an
> >> ager.java +++
> >> b/maven-core/src/main/java/org/apache/maven/classrealm/DefaultClassRealmM
> >> an
> >> ager.java @@ -59,6 +59,16 @@ public class DefaultClassRealmManager
> >>
> >> implements ClassRealmManager
> >>
> >> {
> >>
> >> +/**
> >> + * During normal command line build, ClassWorld is loaded by jvm
> >> system classloader, which only includes + * plexus-classworlds jar
> >> and possibly javaagent classes, see
> >> http://jira.codehaus.org/browse/MNG-4747. +>>
> >> *
> >>
> >> + * Using ClassWorld to determine plugin/extensions realm parent
> >> classloaders gives m2e and integration test harness + * flexibility
> >> to
> >> load multiple version of maven into dedicated classloaders without
> >> assuming
> >> state of jvm system + * classloader.
> >> + */
> >> +private static final ClassLoader PARENT_CLASSLOADER =
> >> ClassWorld.class.getClassLoader(); +
> >>
> >> @Requirement
> >> private Logger logger;
> >>
> >> @@ -329,7 +339,7 @@ public class DefaultClassRealmManager
> >>
> >> throw new IllegalArgumentException( "extension plugin
> >> missing"
> >>
> >> ); }
> >>
> >> -ClassLoader parent = ClassLoader.getSystemClassLoader();
> >> +ClassLoader parent = PARENT_CLASSLOADER;
> >>
> >> Map foreignImports =
> >>
> >> Collections.singletonMap( "",
> >>
> >> getMavenApiRealm() ); @@ -347,7 +357,7 @@ public class
> >> DefaultClassRealmManager
> >>
> >> if ( parent == null )
> >> {
> >>
> >> -parent
Re: maven git commit: better plugin/extensions realm parent classloader
On 2015-01-11 14:48, Hervé BOUTEMY wrote: The change does not affect classloading of "normal" maven runtime, so >there is no need to update the diagram or documentation. IIUC, in the Wiki, what is currently described as "system classloader" is now "initial classloader, containing classworlds", which is the system when using CLI but may be another classloader in other situations where Maven is integrated The change does not affect what is described as "System Class Loader" at all. It only affects the parent of "Project Class Loader" and "Plugin Class Loader" and only when Maven is embedded in other runtimes. The wiki does not document the plugin/project classloader parent and it explicitly avoids talking about embedding scenarios. The change is out of scope of the information provided on the wiki. >Speaking of >which, that documentation is somewhat outdated and does not mention core >extensions realm, for example. I meant to update it some time ago, but >could not find the sources of the diagram. Does anyone know where they are? if you look at attachements, there is a svg associated with the png image Unfortunately, I am not fluent enough in SVG to make the changes (and the file looks generated to me). -- Regards, Igor - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
