On 2009-11-10, at 4:02 AM, Dan Fabulich wrote:
On Friday I was playing cowboy with my experimental thread support,
but here's a more formal proposal around parallel project support in
Maven 3.0.
OUTSTANDING ISSUES
In my earlier revision 833566, I attempted to fix MNG-3004:
"Allow build lifecycle to execute projects in parallel"
http://jira.codehaus.org/browse/MNG-3004
The general consensus around this bug (which has 25 votes) is that
multithreading support can't work correctly right now because of
MNG-2802:
"Concurrent-safe access to local Maven repository"
http://jira.codehaus.org/browse/MNG-2802
Several people have remarked in comments to MNG-3004 that it's
appropriate to try to fix MNG-3004 (parallel projects) without
fixing MNG-2802 (thread-safe local repo). In doing so, we'll allow
users to optionally enable building multiple projects
simultaneously. This is worth doing, because it can be tested in
the wild, and because for some users, it will be immediately useful
(e.g. especially if you use --offline mode).
I agree with these comments, and attempted to implement them in
revision 833566.
MY PROPOSAL
I attempted to check in the code in time for 3.0-alpha-3 a few days
ago. That code was rolled back over the weekend and now lives in the
MNG-3004 branch, because it broke integration tests. All
integration tests now pass on my machine with the MNG-3004 branch,
so I'd like to land it back in trunk again and re-cut 3.0-alpha-3
with this additional feature.
-1
It's not going in that fast. Sorry, but passing all the existing ITs
is the first step and then there is ITs for the feature you multi-
threading feature itself. We simply don't rely on existing ITs anymore
to make sure things work, you must have ITs for this feature itself.
The trunk is now critical for M2Eclipse and anyone else who is
embedding and new features just aren't going in without adequate ITs.
You're not experimenting on users. I'm not trying to detract from any
of the work that you're doing but it's not going in over the course of
a week, and especially not without ITs.
I also don't particularly think it's a good idea without actually
thinking about the impact of not doing MNG-2802.
As I stated on Friday, using the MNG-3004 branch, you can now do this:
mvn install -Dmaven.threads.experimental=4
It works on my machine. If it works for you great; if it doesn't,
we can figure out what's wrong together.
WHY ALPHA-3?
I'd like to land it back in alpha-3 because:
* alpha-2 was in February; I don't want to wait for even four months
to start testing this in the wild
* I did check it in before alpha-3 was released, but it was rolled
back quickly before the release vote was called
* Otherwise I'd want to try to release alpha-4 immediately after
alpha-3, which seems wasteful
I think at this point we've fixing almost everything. So a weekly
alpha would be great, and if you want to get involved with those even
better. But this stuff is not going into the alpha-3 that has been
staged now. We can schedule an alpha-4 for next Monday, provided the
code is merged and has adequate ITs. I don't see any reason why we
can't do weekly alphas.
A NOTE ABOUT DEFAULTS
Currently the code defaults to maven.threads.experimental=1. This
fires up a single executor thread who does all work; the main thread
joins to wait for it to finish. It's also possible to set
maven.threads.experimental=0, in which case everything is done on
the main thread, just like in alpha-2.
If there is consensus, I'd be happy to set the default value to 0,
though that increases the risk that the multithreaded code will get
less coverage in the wild. I think leaving the value at 1 is a good
compromise between avoiding complex threading issues and exercising
tricky code.
WHAT'S IN THE FUTURE?
Post alpha-3 I'm keen on implementing a settings.xml option allowing
users to configure their local repository layout. This will allow
users to choose an alternate implementation that is thread-safe.
Why would the layout have anything to do with thread safety? It should
just be threadsafe don't you think?
Additionally, I think that this feature needs more tooling to make
parallel projects understandable. The logger should be enhanced to
identify which projects are logging which messages, and the final
output of Maven should report more clearly which projects had
warning/error messages, and how many such messages were reported.
What do you say?
All sounds good and the framework has been laid in the ITs where they
can be run with an embedded version of Maven so you can test all sorts
of things now where the same instance of Maven can be used across the
ITs which will allow you to find many threading and share state
problems. Really I would like to see this work fleshed out so that we
can adequately know we're doing the right thing.
This is really much of what Benjamin and I have been doing is laying
the foundation work so that folks can add features safely, but you
just can't whack things into trunk anymore and have the *current* ITs
be the only criterion for being adequate.
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
http://twitter.com/jvanzyl
----------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
