Re: Maven and DevOps

Sat, 14 May 2016 08:07:29 -0700 

Hi there,

thanks for all your feedback.

Just to clarify:
* I do not want to really discuss the processes of mojohaus or maven community projects with this thread. If you get inspired to simplify by this discussion like Tamás (thanks) even better - but mojohaus process discussion should of course take place on mojohaus dev at googlegroups and even for maven dev out of this thread.
* I do not want to blame anybody or say the community did not already do a lot of automation. Thanks for your effort so far and all that has already been archieved.
* I understand that what I was actually proposing is partially covered by releasing SNAPSHOT versions to an internal but pulically available repo. However, see all the discussions on the web that imply that people would like a feature to "relabel" an already build release (e.g. some RC5) as the final release without rebuilding (as nothing has to change and all qualitiy rounds have passed). What I want to do is challenge you to think further and show some potential that could be archived with some improvements in the maven tool ecosystem.
What I wanted to say is that I see still a lot of room for improvement in simplification of deployment:
* Instead of creating a PGP key, registering that key, getting permissions and passwords for deployment repository, logging into nexus after deployment, searching for the staging repository and closing it, voting, waiting, refreshing, publishing, etc. this could all be completely automated by a smart CI chain (what all the DevOps deployment pipelines are about). Deployers would then only require permission to a release branch (or to accept PRs) and the rest would be covered by CI. And this can already be done today with all the tools available. Still traditional versioning, (automatic) tagging, etc. could take place and good old voting processes (on PRs instead) could be kept. This is not what maven or mojohaus seems to aim at but possible. In this context you should also be aware that some releases have been messed up because platform differences (EOL-style, etc.) and a lot of other things can go wrong on the machine of committers causing support effort for the community (still the maven site update fails all the time for me at mojohaus if I build on windows, also for maven-release-plugin I can only build releases with strange workarounds, etc. but all off-topic).
* What I was actually requesting to think about for the tooling is probably more a matter of maven repo tools such as nexus or artifactory. So I guess also the wrong place here. I have seen all the great potential in social and open collaboration. For this reason I would like to see an advanced toolchain that publishes the release of every successful CI build with a technical "version" build from timestamp and potentially the revision of the code repo. That repository should then have social features allowing everybody to give feedback to any artefact. Then the actual releasing could be just a "relabelling" of the technical version to a human readable version. I was thinking of a human readable version just as an alias or "symbolic link" to a technial version.
* Assume maven would habe been designed from the scratch in a way that the "version" is always a timestamp+revision and the human readable version (4.1.0.RELEASE) is just a link to that technical version (just similar to what is done with multiple SNAPSHOTs of the same version today)... I personally think this would solve all the relabelling version issue I stated above.
* My entire vision was to be able to mark an artifact with valuable metadata such as taggig it as bad (insecure, flawed, or whatever) so I could get warnings whenever I use that artifact in my project (instead of OWAPS dependency-check that quadruples the build time). Also a repository could collect a lot of meta-information (how many downloads of each artifact have been done, the maven client could also send an anonymized hash of the groupId of the project requesting the dependency so you collect statistics that can tell how many projects are using a particular artifact, etc.). Assume you could directly see how many projects are using an open source artifact - just one? Or 100? 1000? Quite valuable infos...
Still I hope I could put seeds to some inspiration... I do not expect anyone here to change maven or nexus codebase due to my comments. :)
However, in my business project I leveraged a great potential with an entirely automated deployment pipleline (that also installs the release on test and staging environment after integration tests passed). Many years ago the Open Source teams were sometimes way ahead of what was done in business IT projects but this has IMHO inverted now... 

Cheers
  Jörg

Am 02.05.2016 um 19:06 schrieb Hervé BOUTEMY:

I don't really understand

yes, http://www.mojohaus.org/development/performing-a-release.html is heavy

But if you look at the content, you'll see that:
1. the 3 first sections are about quality expectations, not release,
2. the next 2 are about preparing your environment,
3. then releasing is only the 6th paragraph, with 2 minimal commands to run
release plugin

All the next parts are only about MojoHaus choice to have a staging (like
Apache), with a manual peer review process: if you choose to have another
process, just don't use staging and the previous use of release plugin will
just immediately publish without review

Like Karl Heinz pointed out, there is even the new properties to release
without tagging: but with such an option, tracking what updates go in a new
release is not easy for artifact consumer.
I personnally prefer the SNAPSHOT approach for immediate publication of any
commit: a release is about choosing to make a special publication for more
guaranteed consumption by end-users. And the extra-work is manual since it is
about associated documentation: I really don't see what can be automated in
this part

Then really, I don't understand what to do: we have IMHO 2 types of
expectations, each one with adapted associated efforts
And the long documentation is just because conventions are written down for
the whole team to share: nothing that is about release process.

Do I miss something?

Regards,

Hervé

Le lundi 2 mai 2016 09:51:59 Tamás Cservenák a écrit :

+1 to Jorg's comments ;)

I believe what Jorg says is that process is _manual_:
- do something NOW (merge PR, write email for vote, ...)
- come back after 3 days (and do something more...) (what if you get a baby
in between?)

Ideally, all you'd need is to start a small snowball: merge the PR, and
initiate a release (by filling in a form of some app?). Then, the app could
spin up a vote, that once votes are there, could respond to ML about vote
results, would spin up a release, deploy site, etc.... without any
assistance :)

And during all that you are blissfully looking at another PR :)


Central as collaborative platform... is really great idea, while it is !=
staging at all. Staging is merely meant as QA, not in sense Jorg explains...

Thanks,
T

On Sun, May 1, 2016 at 11:44 PM Karl Heinz Marbaise <[email protected]>

wrote:

Hi Jörg,

On 5/1/16 10:18 PM, Hohwiller, Jörg wrote:

Hi there,

I wanted to share some thoughs I had recently. Maven introduced a
revolution to the Java world and made a really big step for dependency
and build management. Open-Source projects are more productive with

maven.


However, in the last years DevOps showed up and projects start to
continuously build releases in some cases with a fully automated build
pipeline.


When I look at open-source development around I see that we have great
infrastructure with github and pull-requests, etc.

Hm..Apache accepts pull request as well as MojoHaus


But as a downside I also see slow and over-complex processes to get
something released (see e.g.
http://www.mojohaus.org/development/performing-a-release.html - wow that
is not really lean).

Sorry to say that, but you haven't had the opportunity to see over
complex processes...

Simply make a release and open the VOTE for 3 Days (with lazy
census)..what is so complex about that? Yes you need to do some things:

make a release:

mvn -B release:prepare release:perform (gpg signing)

go to target/checkout

mvn site site:stage scm-publish:publish-scm

and send an email for VOTE and wait the result and than click on the
release button for publishing the artifacts to Central...

Hm..?


Furthermore if you don't like it than simply start a VOTE to change
that...What exactly is the problem herer?

I think that those three days are a good trade off between speed and the
options for developers to react cause not all developers are sitting in
front of the keyboards and just waiting someone will start a VOTE (they
are doing other things as well)...


In order not to fingerpoint someone I will pick myself: I got a
pull-request from someone for servicedocgen-maven-plugin that I maintain
at mojohaus. I reviewed the PR and merged it. Unfortunately, I was very
busy then and did not create a release for two month now. It might not
take that much, but still too much. I want to question why do we need
all this stuff and the votings, etc.

Because the community which means the Devs of MojoHaus have decided to
do so...(I'm a member of this as well)...

BTW: Clearly saying to discuss things here on Apache Maven Dev list
which belong to MojoHaus is not a good idea...

For apache the situation is a little bit different, cause there are some
other aspects important...(Like legals etc.)...but no one prevents you
from starting a release....


And yes sometimes mojohaus/apache maven devs are busy...so those are
open source projects ? I don't understand the remarks ?


So assume the following future vision for a maven project:

* When a pull request passed (travis, coveralls, etc.) and gets merged a
CI system automatically builds a release (no need to get PGP keys per
developer just one setup once for the project CI). The release simply
gets a timestamp as version-identifier (yyyyMMdd-hhmmss).

Using PGP etc. is very important in particular related to discussions
about trust and security...(of course only two aspects of that)...

You can remember a feew weeks ago as a single NPM developer could remove
artifacts from NPM which broke thousands of builds? This showed me that
the NPM people don't want to learn from the experiences of Maven central
already had....

Apart from that this would produces a release per commit which usually
not a good idea...CI/CD is the opportunity to make every state a release
...So furthermore you can use staging in repository managers like Nexus
to handle such things?


* Now besides the project being responsible for quality (by having good
tests and only accepting PRs after reasoable review) the community
(artifact users) could also help and do additional quality assurance.

This contradicts to the previous suggestions...

The thing you mentioned...the community is important and must give
feedback which is not always the case or not in the number we wished it
should be....


Assume maven-central would become a collaborative platform where the
users of artifacts could vote and label artifact releases. Add comments,
link CVE or bug reports, etc. Vote +1/-1 on quality or security...

This is more or less already done via the staging before going to Maven
Central...this is what the 72 hours VOTE times is for...

Why should they put that on central...the projects are responsible...
Yes you could imagine this ...but in real life i don't see it...cause
someone needs to maintain the platform Central in this way...who should
do that?

I have large doubts that a users will vote on Central for a release
...cause they are blaming things which do not work (see for example
StackOverFlow) but not filling a JIRA or at least mail on the mailing
list etc.


* Still the project releasing the artifacts could label releases and
associate minor/major release numbers to branches.

Why should we/others associate minor/major release numbers to branches?
Which advantages should that give ?

SemVer.org shows better ideas from the point of view what the artifacts
are doing for users......


In such case however dependencies would not point to a version like
(4.2.1.RELEASE) but instead to 20160501-235901. In order to pick the
right technical version you would lookup the collaborative meta data.
I do not expect everybody to shout hurray to this rought idea. But I
would be happy if people can think about it and may combine it with
other ideas so we get even better in the future some day.

Why should i use something weird like 20160401-232211/20160401-232212
instead of  1.0.0/1.0.1 ? Where is the difference ? What does this mean?

I don't see advantages in comparsion to 1.0.0 / 1.0.1 more the contrary...


See also
https://dzone.com/articles/continuous-releasing-maven

If we are talking about CI/Cd than using release plugin might not be the
best choice...than using versions-maven-plugin etc. are better
alternatives..Furthermore automatic merging in Jenkins etc. can be done
as well...and creating a release can be done within a pipeline without
any problems...


https://devopsnet.com/2012/02/21/continuous-delivery-using-maven/

If you have taken a deeper look during the last months...there are
several activities to make the life cycle more flexible in different
ways...or to be more accurate to make the way open to change things...

For example:
https://issues.apache.org/jira/browse/MNG-5697


I would love to see that maven better supports flexible handling of the
version for the development


  > view while it could simply stay as it is for

the consumer view. When I use variables in versions of POMs (and I do
that in every project today e.g. in combination with
flatten-maven-plugin) I always get warnings from Maven saying:
[WARNING] Some problems were encountered while building the effective
model for ...
[WARNING] 'version' contains an expression but should be a constant.

Than you are using a wrong Maven version...starting from Maven 3.2.1 you
can use things like this in your version:

from the release notes:
http://maven.apache.org/docs/3.2.1/release-notes.html
<quote>
A simple change to prevent Maven from emitting warnings about versions
with property expressions. Allowed property expressions in versions
include ${revision}, ${changelist}, and ${sha1}. These properties can be
set externally, but eventually a mechanism will be created in Maven
where these properties can be injected in a standard way. For example
you may want to glean the current Git revision and inject that value
into ${sha1}. This is by no means a complete solution for continuous
delivery but is a step in the right direction.
</quote>

So you can simply set the version like:

<version>${revision}</version>

and than you do:

mvn -Drevision=1.0 deploy

or

mvn -Drevision=1.0-SNAPSHOT deploy

So where is the problem here?


Still I see no clear picture how maven will adress these needs that
obviously many developer seem to have.

What needs have many developers..?

We could start a discussions about that...and see if we get a clear
picture where the journey could go...


Issues like MNG-4161, MNG-624 and others were simply closed and not
fixed.

MNG-624 would lead to using a version range like JavaScript (NPM) does
etc. and based on the experience we had on version ranges....Furthermore
in the meantime you could use version ranges in parent with next Maven
Release (3.4.0) see

https://issues.apache.org/jira/issues/?jql=project%20%3D%20MNG%20AND%20fix
Version%20%3D%203.4.0


MNG-4161: This can more or less being solved by using
dependencyManagement in parent of the multi module build...Yes not the
optimal solution...but others would need changing the fundamentals of
Maven...and somethings needed to be changed in the pom modell..

Apart from that Benson Margulies has something via Extension: (User
Mailing list):

https://github.com/basis-technology-corp/auto-version-maven-extension


Kind regards
Karl Heinz

  > After just seeing that and levaing an angry comment, I was about

to cancel this mail. However, I just decided to still send it but have
little expectation that it will make any sense...

Best regards

   Jörg

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to