Re: parallelise not overlapping tasks

[Romain Manni-Bucau]

Yes, in theory. In practise they are partially about sequencing cause maven
has nothing else handy (the merge in a multi module project is a headache
for most users i met). Also you cant not respect phases today (regarding my
impl).

Last point is that id the remark was on my last note, this is trivial to
fix by aliasing mojo executions - explicitly or not. The phases are plain
useless since a user will call a mojo, either with its dependencies to
replace phases (mvn +jar for example) or without (mvn jar).

Overall point is that phases now limit maven likely more it helps so for
maven 4 it is something to consider. Short term and for maven 3 i think a
compromise can be found to boost maven builds without requiring the hack to
create 5 modules where 1 is needed - which is current workaround.

Le mer. 2 janv. 2019 18:02, Stephen Connolly <
stephen.alan.conno...@gmail.com> a écrit :

> phases are about the user calling out how far they want to go, not about
> sequencing... at least once we get a propper execution graph
>
> On Thu, 6 Dec 2018 at 13:54, Romain Manni-Bucau 
> wrote:
>
> > Currently maven can't but I expect a way to do it, either in the next xsd
> > as originally proposed or, why not, with a naming convention in the id of
> > the execution (my-exec#after#other-exec or
> something
> > like that if we want it before maven 4)
> >
> > The nice thing is that once done it makes phases pretty much useless (it
> is
> > just about making implicit these dependencies) and it makes the whole
> build
> > parallelizable and not just modules which will often find some bottleneck
> > modules in projects building a distribution.
> >
> > Romain Manni-Bucau
> > @rmannibucau  |  Blog
> >  | Old Blog
> >  | Github <
> > https://github.com/rmannibucau> |
> > LinkedIn  | Book
> > <
> >
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > >
> >
> >
> > Le jeu. 6 déc. 2018 à 14:44, Mickael Istria  a
> écrit :
> >
> > > Hi,
> > >
> > > > any way we move that topic forward beginning of next year?
> > >
> > > I guess providing patches would be the best way ;)
> > >
> > > I'm coming late in this discussion and I'm a newcomer on that list, so
> I
> > > could miss context. This could relate to an effort we are doing in
> > Eclipse
> > > IDE (and Eclipse m2e) to run module builds in parallel. One question I
> > have
> > > is how do you know 2 tasks aren't conflicting? We didn't figure out a
> > safe
> > > way to know that in m2e, maybe I missed something?
> > >
> > > Cheers,
> > >
> >
>

Re: parallelise not overlapping tasks

[Stephen Connolly]

phases are about the user calling out how far they want to go, not about
sequencing... at least once we get a propper execution graph

On Thu, 6 Dec 2018 at 13:54, Romain Manni-Bucau 
wrote:

> Currently maven can't but I expect a way to do it, either in the next xsd
> as originally proposed or, why not, with a naming convention in the id of
> the execution (my-exec#after#other-exec or something
> like that if we want it before maven 4)
>
> The nice thing is that once done it makes phases pretty much useless (it is
> just about making implicit these dependencies) and it makes the whole build
> parallelizable and not just modules which will often find some bottleneck
> modules in projects building a distribution.
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github <
> https://github.com/rmannibucau> |
> LinkedIn  | Book
> <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
>
>
> Le jeu. 6 déc. 2018 à 14:44, Mickael Istria  a écrit :
>
> > Hi,
> >
> > > any way we move that topic forward beginning of next year?
> >
> > I guess providing patches would be the best way ;)
> >
> > I'm coming late in this discussion and I'm a newcomer on that list, so I
> > could miss context. This could relate to an effort we are doing in
> Eclipse
> > IDE (and Eclipse m2e) to run module builds in parallel. One question I
> have
> > is how do you know 2 tasks aren't conflicting? We didn't figure out a
> safe
> > way to know that in m2e, maybe I missed something?
> >
> > Cheers,
> >
>

Re: parallelise not overlapping tasks

[Romain Manni-Bucau]

Hi guys,

I did a quick PoC on a small project. It parallelize the mojo per phase
with a config taken from project properties - not sexy but enables to start
"seeing" something.
You can find it at https://github.com/rmannibucau/parallel-mojo-extension

The main issue it enounters is the fact it can't reuse the ExecutorService
(or its CompletionService wrapper) from the MultiThreadedBuilder - since it
is not done in maven core - and the config which is pretty heavy at the
moment.

However the interesting part is that it is doable, the two main challenges
being:

1. ensure the plugins are safe (not sure threadSafe can be reused, I would
assume that yes but some structure can be not thread safe so not fully sure
and it would be ok while mojo are not parallelize in a single module)
2. find a nice configuration - maybe in terms of dependency between mojo?

The 2 is interesting cause it asks the question of phases which would be
less useful in that context.

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book



Le sam. 8 déc. 2018 à 13:14, Romain Manni-Bucau  a
écrit :

> Using the scm is not enough or it is for single module projects.
>
> You have to have a graph of dependencies (inputs/outputs) and save each
> task state in target to have incremental support
>
> But please note incremental build is != parallel build at mojo level.
>
> This last one is easy to do and a quick win IMHO
>
> Le sam. 8 déc. 2018 00:50, Tibor Digana  a écrit :
>
>> In my projects, the most plugins use single execution.
>> External projects also have this kind of principle.
>> Thus we should have a look in those possibilities where the most plugins
>> can gain the performance.
>> Usually the compiler and tests take long.
>> I know that maven-compiler-plugin:3.8.1 will be incremental which is good
>> of course but we should perhaps continue with gaining the build
>> performance.
>> If somebody has an idea on how to develop a compiler which partially
>> compiles a module depending on SCM changes, feel free to bring it to our
>> mailing list. The same with tests where the set of tests is changed
>> depending on SCM changes.
>>
>> Cheers
>> Tibor
>>
>>
>> On Fri, Jan 19, 2018 at 2:21 PM Romain Manni-Bucau > >
>> wrote:
>>
>> > Hi guys,
>> >
>> > there is no way to parallelize not conflicting tasks for a same phase at
>> > the moment right? Any way it gets under the radar?
>> >
>> > A common example is to run all code analyzis concurrently (findbugs,
>> pmd,
>> > checkstyle, ...) at the same time without waiting for one then the other
>> > etc since all can be very long.
>> >
>> > It could be nice an fancy to define part of the reactor
>> parallelisablity in
>> > the pom, like:
>> >
>> > 
>> > 
>> > process-sources
>> > 
>> > 
>> >   ...
>> >   ...
>> >   ...
>> >   ...
>> > 
>> > 
>> >   ...
>> >   ...
>> >   ...
>> >   ...
>> > 
>> > 
>> > 
>> > 
>> >
>> > anything to enhance it?
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau  |  Blog
>> >  | Old Blog
>> >  | Github <
>> > https://github.com/rmannibucau> |
>> > LinkedIn 
>> >
>>
>

Re: parallelise not overlapping tasks

[Romain Manni-Bucau]

Using the scm is not enough or it is for single module projects.

You have to have a graph of dependencies (inputs/outputs) and save each
task state in target to have incremental support

But please note incremental build is != parallel build at mojo level.

This last one is easy to do and a quick win IMHO

Le sam. 8 déc. 2018 00:50, Tibor Digana  a écrit :

> In my projects, the most plugins use single execution.
> External projects also have this kind of principle.
> Thus we should have a look in those possibilities where the most plugins
> can gain the performance.
> Usually the compiler and tests take long.
> I know that maven-compiler-plugin:3.8.1 will be incremental which is good
> of course but we should perhaps continue with gaining the build
> performance.
> If somebody has an idea on how to develop a compiler which partially
> compiles a module depending on SCM changes, feel free to bring it to our
> mailing list. The same with tests where the set of tests is changed
> depending on SCM changes.
>
> Cheers
> Tibor
>
>
> On Fri, Jan 19, 2018 at 2:21 PM Romain Manni-Bucau 
> wrote:
>
> > Hi guys,
> >
> > there is no way to parallelize not conflicting tasks for a same phase at
> > the moment right? Any way it gets under the radar?
> >
> > A common example is to run all code analyzis concurrently (findbugs, pmd,
> > checkstyle, ...) at the same time without waiting for one then the other
> > etc since all can be very long.
> >
> > It could be nice an fancy to define part of the reactor parallelisablity
> in
> > the pom, like:
> >
> > 
> > 
> > process-sources
> > 
> > 
> >   ...
> >   ...
> >   ...
> >   ...
> > 
> > 
> >   ...
> >   ...
> >   ...
> >   ...
> > 
> > 
> > 
> > 
> >
> > anything to enhance it?
> >
> > Romain Manni-Bucau
> > @rmannibucau  |  Blog
> >  | Old Blog
> >  | Github <
> > https://github.com/rmannibucau> |
> > LinkedIn 
> >
>

Re: parallelise not overlapping tasks

[Tibor Digana]

In my projects, the most plugins use single execution.
External projects also have this kind of principle.
Thus we should have a look in those possibilities where the most plugins
can gain the performance.
Usually the compiler and tests take long.
I know that maven-compiler-plugin:3.8.1 will be incremental which is good
of course but we should perhaps continue with gaining the build performance.
If somebody has an idea on how to develop a compiler which partially
compiles a module depending on SCM changes, feel free to bring it to our
mailing list. The same with tests where the set of tests is changed
depending on SCM changes.

Cheers
Tibor


On Fri, Jan 19, 2018 at 2:21 PM Romain Manni-Bucau 
wrote:

> Hi guys,
>
> there is no way to parallelize not conflicting tasks for a same phase at
> the moment right? Any way it gets under the radar?
>
> A common example is to run all code analyzis concurrently (findbugs, pmd,
> checkstyle, ...) at the same time without waiting for one then the other
> etc since all can be very long.
>
> It could be nice an fancy to define part of the reactor parallelisablity in
> the pom, like:
>
> 
> 
> process-sources
> 
> 
>   ...
>   ...
>   ...
>   ...
> 
> 
>   ...
>   ...
>   ...
>   ...
> 
> 
> 
> 
>
> anything to enhance it?
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github <
> https://github.com/rmannibucau> |
> LinkedIn 
>

Re: parallelise not overlapping tasks

[Enrico Olivelli]

What about having parallel 'planes' of execution?
Stuff like checktyle, rat, validation plugin may run in their own plane of
execution.
By default every existing plugin will be on a 'default' plane.
When the build starts we start a thread/fork a process for each plane used
by plugins declared in the pom.
Maybe we could define standard planes so that plugins will be able to
choose from a well know list of ids.

This approach is very naive, because it does deal with a real graph, but it
can be an easy step compared to a global refactoring/ introduction of
input/output declaration for each existing plugin.

This can be orthogonal to phases, each plane will execute every of the
phases.

Maybe I have a limited view of the Maven core model.
Hope that helps

Enrico

Il gio 6 dic 2018, 15:35 Romain Manni-Bucau  ha
scritto:

> Mojos already have the ability to say if they are threadsafe, what do you
> see missing?
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github <
> https://github.com/rmannibucau> |
> LinkedIn  | Book
> <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
>
>
> Le jeu. 6 déc. 2018 à 15:23, Mickael Istria  a écrit :
>
> > I think there is a difference between scheduling tasks (one is after the
> > other) and assuming that 2 tasks that are ready can run in parallel.
> > Adding scheduling data would be helpful and get rid of the concept of
> > phases, for sure; but we also -and more importantly- need the existing
> > mojos to be audited and be able to declare whether they're thread-safe
> > before running them in parallel, and this sounds like a gigantic work.
> >
>
-- 


-- Enrico Olivelli

Re: parallelise not overlapping tasks

[Mickael Istria]

Ok, my bad, thanks for the hint.

Re: parallelise not overlapping tasks

[Romain Manni-Bucau]

Mojos already have the ability to say if they are threadsafe, what do you
see missing?

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book



Le jeu. 6 déc. 2018 à 15:23, Mickael Istria  a écrit :

> I think there is a difference between scheduling tasks (one is after the
> other) and assuming that 2 tasks that are ready can run in parallel.
> Adding scheduling data would be helpful and get rid of the concept of
> phases, for sure; but we also -and more importantly- need the existing
> mojos to be audited and be able to declare whether they're thread-safe
> before running them in parallel, and this sounds like a gigantic work.
>

Re: parallelise not overlapping tasks

[Mickael Istria]

I think there is a difference between scheduling tasks (one is after the
other) and assuming that 2 tasks that are ready can run in parallel.
Adding scheduling data would be helpful and get rid of the concept of
phases, for sure; but we also -and more importantly- need the existing
mojos to be audited and be able to declare whether they're thread-safe
before running them in parallel, and this sounds like a gigantic work.

Re: parallelise not overlapping tasks

[Romain Manni-Bucau]

Currently maven can't but I expect a way to do it, either in the next xsd
as originally proposed or, why not, with a naming convention in the id of
the execution (my-exec#after#other-exec or something
like that if we want it before maven 4)

The nice thing is that once done it makes phases pretty much useless (it is
just about making implicit these dependencies) and it makes the whole build
parallelizable and not just modules which will often find some bottleneck
modules in projects building a distribution.

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book



Le jeu. 6 déc. 2018 à 14:44, Mickael Istria  a écrit :

> Hi,
>
> > any way we move that topic forward beginning of next year?
>
> I guess providing patches would be the best way ;)
>
> I'm coming late in this discussion and I'm a newcomer on that list, so I
> could miss context. This could relate to an effort we are doing in Eclipse
> IDE (and Eclipse m2e) to run module builds in parallel. One question I have
> is how do you know 2 tasks aren't conflicting? We didn't figure out a safe
> way to know that in m2e, maybe I missed something?
>
> Cheers,
>

Re: parallelise not overlapping tasks

[Mickael Istria]

Hi,

> any way we move that topic forward beginning of next year?

I guess providing patches would be the best way ;)

I'm coming late in this discussion and I'm a newcomer on that list, so I
could miss context. This could relate to an effort we are doing in Eclipse
IDE (and Eclipse m2e) to run module builds in parallel. One question I have
is how do you know 2 tasks aren't conflicting? We didn't figure out a safe
way to know that in m2e, maybe I missed something?

Cheers,

Re: parallelise not overlapping tasks

[Romain Manni-Bucau]

Hey guys,

any way we move that topic forward beginning of next year?

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book



Le lun. 22 janv. 2018 à 01:04, Tibor Digana  a
écrit :

> I am facing very slow Maven reporting/site.
> I have optimized JavaDoc and avoided the javadoc plugin downloading
> sources, Java API which has improved the performance but still Site takes 8
> minutes at least to complete.
> It would be worth to mark some reporting plugins running in parallel in
> aggregator POM.
>
> On Fri, Jan 19, 2018 at 2:20 PM, Romain Manni-Bucau  >
> wrote:
>
> > Hi guys,
> >
> > there is no way to parallelize not conflicting tasks for a same phase at
> > the moment right? Any way it gets under the radar?
> >
> > A common example is to run all code analyzis concurrently (findbugs, pmd,
> > checkstyle, ...) at the same time without waiting for one then the other
> > etc since all can be very long.
> >
> > It could be nice an fancy to define part of the reactor parallelisablity
> in
> > the pom, like:
> >
> > 
> > 
> > process-sources
> > 
> > 
> >   ...
> >   ...
> >   ...
> >   ...
> > 
> > 
> >   ...
> >   ...
> >   ...
> >   ...
> > 
> > 
> > 
> > 
> >
> > anything to enhance it?
> >
> > Romain Manni-Bucau
> > @rmannibucau  |  Blog
> >  | Old Blog
> >  | Github  > rmannibucau> |
> > LinkedIn 
> >
>

Re: parallelise not overlapping tasks

[Tibor Digana]

I am facing very slow Maven reporting/site.
I have optimized JavaDoc and avoided the javadoc plugin downloading
sources, Java API which has improved the performance but still Site takes 8
minutes at least to complete.
It would be worth to mark some reporting plugins running in parallel in
aggregator POM.

On Fri, Jan 19, 2018 at 2:20 PM, Romain Manni-Bucau 
wrote:

> Hi guys,
>
> there is no way to parallelize not conflicting tasks for a same phase at
> the moment right? Any way it gets under the radar?
>
> A common example is to run all code analyzis concurrently (findbugs, pmd,
> checkstyle, ...) at the same time without waiting for one then the other
> etc since all can be very long.
>
> It could be nice an fancy to define part of the reactor parallelisablity in
> the pom, like:
>
> 
> 
> process-sources
> 
> 
>   ...
>   ...
>   ...
>   ...
> 
> 
>   ...
>   ...
>   ...
>   ...
> 
> 
> 
> 
>
> anything to enhance it?
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github  rmannibucau> |
> LinkedIn 
>

Re: parallelise not overlapping tasks

[Romain Manni-Bucau]

Hi Robert,

this is the gradle way but it only works because you also have the
dependency chain between plugins in gradle. This is missing in maven (like
group:artifact@execution before group':artifact'@execution' / or after) and
therefore can't be worked out at all with just inputs outputs for backward
compatibility - and if you assume all plugins will migrate, this feature
would be available for end users in a very long time which would be sad.
That is why I thought letting the user helping providing some insights.

Anyway, being able to parallelise the code analyzis, front and back builds
for modules having frontend-maven-plugin + standard java in the same module
etc... would be very beneficial if we can have something working with maven.





Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn 

2018-01-19 15:08 GMT+01:00 Robert Scholte :

> Hi Romain,
>
> I don't think this is something for the pom.
> It is the plugin which should know if it has effect on the build.
> Or maybe more global: if we could define for the plugin what is input and
> what is output, then it should be possible to decide which parts could run
> in parallel.
>
> thanks,
> Robert
>
>
> On Fri, 19 Jan 2018 14:20:50 +0100, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
> Hi guys,
>>
>> there is no way to parallelize not conflicting tasks for a same phase at
>> the moment right? Any way it gets under the radar?
>>
>> A common example is to run all code analyzis concurrently (findbugs, pmd,
>> checkstyle, ...) at the same time without waiting for one then the other
>> etc since all can be very long.
>>
>> It could be nice an fancy to define part of the reactor parallelisablity
>> in
>> the pom, like:
>>
>> 
>> 
>> process-sources
>> 
>> 
>>   ...
>>   ...
>>   ...
>>   ...
>> 
>> 
>>   ...
>>   ...
>>   ...
>>   ...
>> 
>> 
>> 
>> 
>>
>> anything to enhance it?
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github <
>> https://github.com/rmannibucau> |
>> LinkedIn 
>>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>
>

Re: parallelise not overlapping tasks

[Robert Scholte]

Hi Romain,

I don't think this is something for the pom.
It is the plugin which should know if it has effect on the build.
Or maybe more global: if we could define for the plugin what is input and  
what is output, then it should be possible to decide which parts could run  
in parallel.


thanks,
Robert

On Fri, 19 Jan 2018 14:20:50 +0100, Romain Manni-Bucau  
 wrote:



Hi guys,

there is no way to parallelize not conflicting tasks for a same phase at
the moment right? Any way it gets under the radar?

A common example is to run all code analyzis concurrently (findbugs, pmd,
checkstyle, ...) at the same time without waiting for one then the other
etc since all can be very long.

It could be nice an fancy to define part of the reactor parallelisablity  
in

the pom, like:



process-sources


  ...
  ...
  ...
  ...


  ...
  ...
  ...
  ...





anything to enhance it?

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  
 |

LinkedIn 


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

parallelise not overlapping tasks

[Romain Manni-Bucau]

Hi guys,

there is no way to parallelize not conflicting tasks for a same phase at
the moment right? Any way it gets under the radar?

A common example is to run all code analyzis concurrently (findbugs, pmd,
checkstyle, ...) at the same time without waiting for one then the other
etc since all can be very long.

It could be nice an fancy to define part of the reactor parallelisablity in
the pom, like:



process-sources


  ...
  ...
  ...
  ...


  ...
  ...
  ...
  ...





anything to enhance it?

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn