I agree and disagree... but I 'm gonna shut up.
I think you guys are moving in positive direction with mvn...
i'm just pissed off that the recent changes to specs has derailed my
automation efforts.
I certainly did not mean for this email to turn into a mvn ranting
session.
and maybe tomorrow I can cope better with this and provide a
reasonable response to this email.
I appreciate the work you guys have been doing... and while not
perfect, IMO it is positive direction.
and I will leave it at that for now.
thanks for your time... sorry if I ruffled some feathers.
--jason
On Dec 15, 2006, at 11:05 PM, Jason van Zyl wrote:
On 16 Dec 06, at 12:10 AM 16 Dec 06, Jason Dillon wrote:
On Dec 15, 2006, at 7:51 PM, Jason van Zyl wrote:
IMO the remote repos are for user connivence *ONLY*,
That is horse shit. What they contain is a product of what you
people put in them. You guys for example have something that is
tagged, which you released and which is not reproducible. That's
not Maven's fault, that's your fault. You are hosing anyone
trying to consume your stuff. And if people are doing that to you
then your should irate like anyone should be who looks at your
tag for 1.1.1. That's why the release plugin behaves the way it
does. It screamed at you when you tried to release that 1.1.1 tag
I bet and you side stepped what you should have done and did
something manually.
First off... this is *my opinion*... not sure how you can jump to
the conclusion that my oppinon is horse shirt... or any mammal
shit for that matter. But I have been know to say that mvn is
crap many times before... so if you feel better stating that my
opinion is shit... well, then go for it.
Second, what assurance does any project have any any given
artifact in the central repo will remain there asis for the
foreseeable future?
It's always been our policy that artifacts that we place in the
repository are not removed. From syncing partners like Codehaus,
Apache, ObjectWeb, or Mortbay they have to tell us that they do not
want us to match the source they give us. We have never culled the
repository.
There are are already situations where bits have been added and
removed (at least one which I requested to get around a sync
problem from ASF) and a few others which I have heard about
through others.
Only if the repository is diddled on the source side. We cannot
control in all places what is done. You guys seem to manually
delete artifacts from this side which wreaks havoc.
There is no audit trail for central or any other popular mvn repo,
so any release manager with half their wits is going to think
twice about trusting any content which is pulled from it.
Two separate issues. People behind firewalls generally do not and
cannot use the central repository directly but they manage ones
that do rely on central. And they can for releases because we do
not delete anything. Everything release be signed and that will get
easier to do automatically so though it's not fully automated to
check the signatures you can verify what you have. We will have a
transition period where unsigned artifacts will be accepted but
shortly anything coming in by any means will have to be signed and
the metadata will carry with it a reference to its source.
one critical error I believe that everyone is making is thinking
that deployed artifacts are permanent...
Deployed releases that we place in the repository are most
certainly permanent.
What assurance do I have that artifacts are going to exist in the
repo in 10 years? 20?
That is our policy and we will bolster our infrastructure but
Ibiblio gives us the assurance that what we give them stays in
perpetuity. That's their policy. That's why we still mirror things
there even though we are using our own dedicated box. I also have
backups from day one that are stored in an offsite along with my
other backups.
How about the assurance that they have not been tampered with?
Signatures that you are supposed to use for releases here since
always. We don't enforce that yet but anything coming from Apache
should have them.
Digest files don't do jack... as if someone alters an artifact,
and I download it into an empty repo... then it all looks fine to
the consumer.... and its highly regular for users to nuke their
repo when problems happen... and problems are regular too.
It will not be long before it will simply be mandatory to have a
PGP key sign your goods if they are going to make it into the
repository.
Snapshots can be transient depending on the policy of the
repository. What I've been told is that infrastructure is telling
you to remove old versions from the repository at Apache which
can have disasterous effects. Your artifacts deployed to central
should not disappear so I'll go make sure that we're not deleting
files that have been removed from the source. But we generally
assume when we are syncing from a source that the source knows
what it's doing. If you delete stuff on this end then we expect
you to understand what effect that will have on your clients.
Many users may understand, but there is always one that will not,
which throws off the entire system.
Which users? You as a user supplier of artifacts populating your
repository?
If one weak link decided to re-release version 2.0 of something
that effectively breaks compatibility with consumers of the
previous 2.0 release, then then entire system is compromised...
who is to say that any other artifact might just change under the
covers.
There are lots of the issues we know about and are fixing, lots of
stuff in the wiki about maven-artifact changes in 2.1. You're not
saying anything yet we haven't heard before. That's not really
germane to this your issues anyway. You can't even get non-SNAPSHOT
so it wouldn't matter if it was secure, it would still be useless.
Granted that does not happen often, but it does happen, and will
most certainly continue to happen.
It's happened once that was very noticeable which was the overlay
of a binary incompatible version of commons-logging 1.0 was laid
down. Using the release plugin helps and when we make a deployment
transactional we can enforce not being allowed to deploy the same
version again. So it's not like we're sitting on our laurels, we've
seen these issues and over time we add measures.
Maybe its possible to remove the chance of it happening from
central, but projects do not just depend upon central... its easy
enough to setup a repo, then include that into your project. And
the managers of that repo may remove add/change artifacts at
will. So my comment about the transitive nature of all mvn repos
is much more general... and certainly not mean to know mvn, but
more as a warning that artifacts on remote repos are much more
transient that many people (like you) believe they are.
Then don't use those repos, or label them as snapshot repos. As far
as Geronimo is concerned why do you need anything more then central
as a source? Aside from your SNAPSHOT dependencies.
This will only stop when Archiva is in full effect. The only way to
submit anything to central will be via Archiva. Any project who
wishes to have the same stability will only take artifacts that
have passed through and instance of Archiva. You'll know you're
using an instance of Archiva because we'll have a wagon for doing
that and it will be configured. It will eventually be the default.
It will simply be the Grizzly client and Jetty using the Grizzly
connector.
which they are most certainly not, so we should generally always
build from source to ensure that everything is being included at
the right version.
No you shouldn't. That defeats the entire purpose of Maven.
Certainly not Jason...
Yes, building from source when not required defeats the purpose of
Maven. It is generally not required.
and I'm surprised to hear this from you as I have heard you talk
about the plug-ability of build functionality as much more
important to the purpose of maven than the artifact remoting. IMO
the plugin framework of mvn is much more important to the purpose
of mvn than remote artifact handling... and more so the lossy
artifact handling is more of a detriment to mvn than anything
else. I'd like to throw the remote artifact capabilities into the
trash... and then we'd find a reliable and trustworthy build
platform in mvn.
The remote artifact handling is one of Maven's greatest strengths.
It relies on the integrity of the repository and when everyone
expect us, the Maven developers, to clean up their shit it's not
going to work. If you don't want to play nice with everyone else
then stick your repository in SVN and be cut off from everyone
else. When the central repository gets more robust through the use
of Archiva it will become an invaluable resource. it's far more
important then you think.
Projects are not accountable for stuff they put in repositories is
what make the public space unreliable in its default mode. Maven
entirely reliable when the repositories you use are and many people
do maintain their own repos. That is the norm for corporate usage.
If you also really really want to you can use the SCM Wagon which
works off an SCM. The unreliability factor is almost always the
over use of SNAPSHOTs and people being cavalier with repositories.
Maven will only be able to improve this over time when the
repository management tools come into effect. It's not a small
problem to try and make this all work and unfortunately we've
determined people are too negligent, lazy, and don't think it's
important to maintain the repositories even though they have become
a critical resource.
I can't even count the number of times that mvn builds have broken
due to external dependency changes... no local sources changes,
but come in to the office the next day and the build is just fucked.
SNAPSHOTs. Nothing with a SNAPSHOT is reliable. And by that the
"SNAPSHOT" identifier. It is only an identifier, on the remote side
there are no artifacts with "SNAPSHOT" in them. That identifier
maps to a timestamped version. So if you wanted reliability from an
ever changing mass, once you had a build that worked you could
write a plugin that passed over them all, looked up the last
timestamp from the metadata in the repository and plug those in.
You have zero guarantee of a working build with a SNAPSHOT. None.
Never have, never will.
You have unmaintainable and non-reproducible builds because you
have such a massive number of SNAPSHOTs and you never make any
incremental releases. What do you expect when you're building
upon sand?
Ya could be... mvn's snapshot mechanism is more than enough rope
for a team to hang them selves 10x over. Is that the teams fault
or mvn's fault for handing over the tools to hang themselves?
That's your fault, SNAPSHOTs mean unstable by their very
definition. You don't have to use SNAPSHOTs it's a convenience for
change __within__ your project.
Probably a mix of the two. Which I why I tend to try to limit
the usage of that garbage... but since its in mvn, and other folks
just assume that if its there it must be good, they tend to use it...
I don't who told you to use SNAPSHOTs but they are bad news when
you are using an external project SNAPSHOTs. You have squat in
terms of reliability. Use all non SNAPSHOT versions, and central
and if you have a breakage it's guaranteed to be in your project.
Period. Can't be anything else. The only time really bad things
have happened is the accidental commons logging thing and when we
have failures at places like Codehaus, and Ibiblio. We've moved to
fix that by using Contegix, trying out S3 and trying to get more
mirrors. Again, we're not resting on our laurels.
but more its also that g is a different type of project that has a
bunch of dependencies on external projects, which are moving at
similar (if not faster) speeds... and IIUC this is what SNAPSHOT
artifacts are meant to address, though I think that the actually
implemented, that with inconsistent deployment of snaps, has
completely failed as a solution.
Then you're really one project if you're so highly coupled and
maybe you should just put it all together. I think that's a clear
indication of a problem.
for example, G server depends on OpenEJB... but who knows when the
latest snap was deployed for OpenEJB, so to ensure that your G
server build actually works you *MUST* build OpenEJB from source.
No it doesn't. Once you have a build that works, you can use the
timestamp as the version and stop letting it swing. Ask OpenEJB to
deploy everything to one repository and have no separate SNAPSHOT
repository so that they won't be nuked. If you cannot stop using a
SNAPSHOT something is wrong.
This is not a mvn problem by itself, but a process problem for not
getting regular snaps deployed. But assuming that there was a
nightly deploy of all that stuff, then there is still a window
during the day when the code could have changed in OpenEJB and G
server which would not be picked up due to the SNAPSHOT not being
deployed until hours later...
Absolutely, don't use them all the time. They have to do this
periodically to give you something stable.
and then even once it was deployed, the default policy of waiting
daily before updating snapshots... basically makes a huge window
of times when a user might get the wrong code and end up with a
failed build.
THE ONLY WAY TO SOLVE THIS IS TO BUILD IT LOCALLY.
No it's not. I think you have a fundamental misunderstanding about
how versioning works in Maven. At least in Maven 2.x. OpenEJB can
deploy as many snapshots as they like, you have found one to work.
You have found a set of snapshots to work then lock them down.
This only happens for a small number of deps, where there is tight
coupling... but it does happen... and since G server and OpenEJB
are so intimate with each other, we run into this all of the
time. And there really is no solution to solve the problem, short
of decoupling them, or building from source.
And since the chances of these being decoupled anytime soon is
slim to none... we really have no choice but to build openejb2 as
a dependency.
That's simply nuts. That you have to do that means you're screwed.
That you, at no point in time, can get a stable release is not
good. Everyone needs to across your project must settle on a
version as the sign post.
Get the producers of your artifacts to release things more
frequently, this ensures that SNAPSHOTs are removed from the
chain. I look in your server build and you've got SNAPSHOTS
**all** over the place. OpenEJB, ActiveMQ, Tranql, your spec
JARs. You control, or have relationships with most of these
projects.
Give me a break dude... we have relationships with Maven too...
but have had little to no luck in getting some critical bugs fixed
which affect our builds, like how we have to split up our build
into two stages to avoid the bug of extensions that are built in
the same cycle from being picked up.
Like the one Brett applied last week for you? LIke the RAR plugin
that was put into the cycle for you? Like the XMLBeans plugin I
have tried to get released for you? LIke the plugins I'm making to
try and make staging easier and licensing easier that Dain used
yesterday to make your release?. I took your GPG plugin and
integrated it into Maven plus I field the bitching and fix shit for
people all the time. It doesn't get all fixed at once. But I deal
with this shit all time where Maven is getting pegged for shit that
the tool is not responsible for. There are lots of bugs because we
have tons of users and they find shit all the time. I could give a
rat's ass what anyone thinks because we're honestly can't do
anymore then we already are.
The same shit happens for other projects too. G depends on a lot
of other projects, and its non feasible to wait for our dependency
projects to deploy the right versions for use to move forward all
the time.... otherwise we would be waiting forever for other
groups to move.
A timestamp will do as a version. Always has in Maven 2.x. You
don't need a final release. You need something that will not change
out from underneath you which a SNAPSHOT is meant to do. So you have:
http://snapshots.repository.codehaus.org/org/codehaus/swizzle/
swizzle-jira/1.3.1-SNAPSHOT/
Now if you look in:
http://snapshots.repository.codehaus.org/org/codehaus/swizzle/
swizzle-jira/1.3.1-SNAPSHOT/maven-metadata.xml
You will see that 20061122.035631 was the actual name of the JAR
that corresponds to the latest SNAPSHOT. That will work as a
version that will not change. For all intents and purposes that
SNAPSHOT should stick around long enough for you to get a release.
For example... if we wanted for your team to fix MNG-1911 before
we finished our m2 conversion... well, we'd still be using m1
today... and probably for the foreseeable future.
Well, what does any software project do given many demands and
limited resources? It's got 0 votes. Scream louder. I use them to
gauge what I do in plugins:
http://repo1.maven.org/reports/plugins/plugin-issues.txt
What else are we supposed to do with the unsatisfied multitudes?
That... and the fact that mvn releases each damn little part of a
project severally forces each of these projects to go through
legal hell to make sure that everything is up to policy with legal
files etc... which normally would only have applied to source
files and to the finally distribution.... but mvns opened that up
to include almost every single damn file that is part o a build...
which is ridiculous.
Modularization is a good thing. Crappy problems like the legal
problem can be solved when the group of Maven users needs
something. That crappy legal problem has been fixed and we're
testing it and it worked for your release yesterday. Dain used it
and you don't have to copy any legal crap into every JAR. It's done
transparently now, or will be when you inherit from the ASF POM:
http://idisk.maven.org/jvanzyl/Public/RemoteResourcesHandling.png
That one I listened to and fixed.
You don't wait until it's time to release to get the producers of
your dependencies to crap out some junk you can use. Get them
release, lock your stuff down to it, move on. Repeat. You should
have no SNAPSHOTs other then ones for your own projects in your
POMs. If everything is changing everywhere it's total insanity
This is not always a possibility... if any of these groups
releases there goods a slow or you our we do, then nothing is
going to move. Using SNAPSHOTS, or building from source is really
the only thing that is allowing us to move forward, because there
is so many changes going into each of these components which are
required for G to function. Its simply not possible to lock step
our progress with the release schedule of other projects, we must
for development take incremental snapshots of code.
That's fine but you still can lock down on a timestamp. You have to
at some point draw the line. The second you lock one thing it
forces others to lock down. You declare that for this week using
this version of OpenEJB, everyone must follow along and you prepare
for the following week to use the SNAPSHOT built on X day. Everyone
trying to look at sources to match signatures is simply not
scalable. Locking down to a version, even if it is a timestamp, is
your way of communicating "you're going to have to coordinate with
others to align things and live with with for the week". Things
cannot change so radically that everything breaks. People have to
use feature branches or be more careful about what they are
expelling. Additions should not affect consumers of artifacts, and
if it is only additions (which I assume because you're all
following JEE standards) then you have to plan that better for
consumption. If people are breaking binary compatibility five times
a week they should be tarred and feathered:
http://wiki.eclipse.org/index.php/Evolving_Java-based_APIs
You can use CLIRR to nail offenders:
http://clirr.sourceforge.net/
And look at that ugly Maven site and you know there's a plugin.
Build everything from source is not something you should be doing
because you should not have to. It is one of the most obvious
signifiers that you are doing something very wrong.
I do not meant to build everything from source... though if I did
feel that was needed, it is most certainly not a sign of doing
something wrong. If I wanted to know exactly what changes went
into the system... or wanted to know what changes affected which
tests, then that is really the only way to tell.
Again it's not. You have crappy releases and crappy planning. You
have too much change to absorb reliably at once and is unsustainable.
I'm not suggesting that we want to build commons-logging or howl
all by hand... but for the projects that change the most (those
which we are using SNAPSHOTS for), then for the automated scenario
is it ideal ti build from source, and eliminate any guess work
that might come in to play when using the lossy mvn SNAPSHOT
downloading mechanism.
You're using the SNAPSHOT mechanism incorrectly. It's for internal
use on a project and external dependencies a rarity.
This is especially key, when build take a while to perform, and
during the time it takes to run a longer build that one or more
snapshot builds of a dependency are created.... unless the
automation system actually tracks the exact artifacts produced and
hands them to the target build, there is no way that any
correlation from build to build can be done. This is a CRITICAL
flaw in the mvn SNAPSHOT mechanism.
No it's not you use it incorrectly. You always have the exact
version number for a SNAPSHOT. Always.
We don't need to guess what changes were include in a build... but
for large projects and distributed builds... mvn offers no
assurance to what artifacts are actually used.
You can always resolve the exact version. You guys have to many
which would make it annoying to look in the repo to find them. But
we could easily make a tool.
You have way too many points of variation and all your projects
seem to be in constant swing with one another. If that's the case
it begs the question whether they really separate and why not
just lump them all together if they are so coupled together.
Which generally the case when you cannot eliminate SNAPSHOTs from
your build. When you see SNAPSHOT not from your project in our
POM it is a warning sign.
Dunno about that... the most tightly coupled SNAPSHOT we have now
is OpenEJB... and really, I think that its so tightly coupled that
it really should be in the same codebase. BUT, that is not what
other key plays believe, so then we are left trying to work the
build system to generate something that is predictable given the
circumstances. Unfortunatly, mvn does not have a good answer for
this problem... all it has is SNAPSHOT... and IMO that is not
worth a damn.
Not the way you use it because that's not what it was intended for.
Its more trouble than it is worth... and more so since all you
people are using it everywhere, its very difficult to educate
people to its harm.
I don't think people use it the way you do. Most people seem to
understand it's for dealing with variation on the project you are
working on. And that SNAPSHOTs typically occur within your own
reactor.
You guys seem to want to use the release plugin for releasing
things with SNAPSHOTs in them. That's not what it was made for
and again it's a function of you not managing your relationships
with your dependent projects. A release with SNAPSHOTs is not a
release. Tagging something that contains a POM with a SNAPSHOT
identifier is completely useless. I mean look at your 1.1 release:
http://svn.apache.org/repos/asf/geronimo/server/tags/1.1.1/
That's insane. That's a tag! How did you do a release with
SNAPSHOTs? That's not Maven's fault if you can't built that again
if you were careless enough to tag something with SNAPSHOTs. You
have got to use releases of plugins, parent POMs, and dependences
more frequently. Maven has lots of bugs, sure, but this mess is
largely your own doing. There are lots of Maven users with way
more source code and modules then you and don't have nearly the
number of problems you do.
I had nothing to do with any of that release... I have been trying
to unfuck a bunch of issues with the release process in
general.... and for the record its not all mvn related stuff that
needs unfucking. Its more policy on how one uses mvn that needs
work... some issues with the default plugins, some bugs, but more
than anything it more the policy about how one uses mvn to build/
release projects rather than mvn itself.
The stuff done of late for resources will help everyone at Apache.
The legal and signing nonsense will be taken care and I will take
measures to clean up the repository fuckups.
I really feel that the little short-cuts and side-steps around
maven problems are going to kill us. And really, I don't see
how any medium to large sized group can effectively use maven to
manage releases of their projects...
Lots of projects do it fine. Your dependency chain is fragile
which is the problem. Your coupling is too high which will not
let you do releases as you have to, or you are not working with
the producers of your dependencies to create releases. In trunk
right now you have a SNAPSHOT for the POM that seems to contain
the dependencies for everything you need. That's wrong. You need
to release those parent POMs and the dependencies in them. It is
natural to have SNAPSHOTs for the project in question, for CI and
while you're developing. But a SNAPSHOT in a dependency means
that you, as a last resort, had to use something in flux because
it cannot work otherwise. This immediately should signal you to
contact that project to fix it, and release it. And again,
because all your dependent projects seem to share many of the
developers you should be able to do that.
Lots of projects do fine... sure. Smaller projects with less
requirement for durable/repeatable and stable builds should work
fine. But larger projects, with more moving parts that need to
have builds repeatable over 10 years+ (with no changes to tags)
will almost certainly fall flat on their faces when using mvn...
UNLESS certain precautions and limitations are heeded.
I'm building several projects way bigger then yours for mission
critical systems that have governments involved if they don't work
and it builds fine, and it will build fine in 10 years.
The same is true for almost any other build tool... make and ant
both can end up in the same situation... BUT... the key difference
is that mvn makes it way to easy to get into a situation where
your builds are non-durable and non-stable right out of the box.
Its got instabliltiy and non-durability built right into it with
its remote repository... a problem which make and ant do not
suffer from... and a reason why ant is more prevalent in
commercial shops than mvn is.
That's just not true. You seem to fundamentally misunderstand the
use of a SNAPSHOT. And a build from 3 years ago running against
Ibiblio done as a proper release will build. In Norway more people
use Maven :-) And more and more people are using Maven. Our stats
are going through the roof and repository usage rate goes up all
the time. Corporate users also manage their own repositories so
they don't have any problems.
* * *
Anyways, I'm gonna stop here. I could certainly go on and respond
to everything in this mail... but I don't see the point.
My purpose was not to diss mvn here... but more so to suggest that
there are more effective ways to use mvn which fit better into our
integration model... there are better ways to use mvn ASIS with
the requirements/restrictions we have now.
But even more so... my point was... that changes that have been
made recently have really fucked over the work I have been doing
to produce automated reliable durable builds for our projects...
and yes, based on all those SNAPSHOTS of all those projects.
I had it working and it was working well... until the model of
versioning specs changed... more specifically since the specs/
trunk branch could not actually build and now it builds artifacts
which are not being using by anything... the bits that are needed
are now only available to build off of little spec module
branches, which is a huge pain to automate.
IMO the only problem specific to maven here.... is that this
versioning model was allowed by maven and even more so promoted by
mvn since that is how you folks manage your plugin modules. So
we have the lemmings following by example... right off the cliff.
And lastly... none of this is anything I consider mine... or my
own... its all junk that I have picked up and been trying to fix.
And at one point I did believe that mvn was going to help me do
that... in retrospect... Ant would have been a better tool....
less bugs... less mystery... less time trying to figure out just
what the fuck broke.
I doubt it. When it came down to releases and updating artifacts
and trying to tie everything all together. Lots of people might
have problems but even now I bet there are people who could help
with the build if necessary. If you did it in Ant you would be the
only person who would know how it worked. You would get the nice
Inner Platform Effect with a build your size:
http://thedailywtf.com/forums/69415/ShowPost.aspx
I'd bet my life you would have more overall problems using Ant.
Just because you could get it to work doesn't mean it would scale
or be something anyone else could comprehend. It's probably already
hard enough with what you have.
I cant' even count the number of times I have had to nuke my local
repo, rebuld specs, openejb, g by hand to debug someone else's mvn
problem... even more prevalent is the number of times I have heard
of other people doing the same. We spend more time fucking with
mvn than we do actually writing code for Geronimo... and that is
certainly a sign that something is wrong.
Yes, there are a handful of problems with maven-artifact that cause
some grief. Can't fix everything all at once.
--jason
