I though I should share two scripts I hacked together when working with multiple aggregation roots from different GIT repositories...
The first one I call `mvn-release-all.rb` #!/usr/bin/env ruby require 'nokogiri' want_color = `git config color.ui` $color = case want_color.chomp when "true"; true when "auto"; $stdout.tty? end def red s; $color ? "\033[31m#{s}\033[0m" : s end def green s; $color ? "\033[32m#{s}\033[0m" : s end def yellow s; $color ? "\033[33m#{s}\033[0m" : s end def cyan s; $color ? "\033[36m#{s}\033[0m" : s end def grey s; $color ? "\033[1;30m#{s}\033[0m" : s end def purple s; $color ? "\033[35m#{s}\033[0m" : s end def read_pom pom_file Nokogiri::XML(File.open(pom_file)) { |config| config.nonet }.remove_namespaces! end def group_id pom_doc (pom_doc.xpath('/project/groupId').first or pom_doc.xpath('/project/parent/groupId').first).text end def artifact_id pom_doc (pom_doc.xpath('/project/artifactId').first or pom_doc.xpath('/project/parent/artifactId').first).text end def version pom_doc (pom_doc.xpath('/project/version').first or pom_doc.xpath('/project/parent/version').first).text end def upstream_version pom_doc, coord g, a = coord.match(/(^.*):(.*)$/i).captures result = [] pom_doc.xpath('/project/parent').each do |n| pg = n.xpath('groupId').first.text pa = n.xpath('artifactId').first.text pv = n.xpath('version').first.text if n.xpath('version').first if g == pg and a == pa and pv result << pv end break end pom_doc.xpath('/project/dependencyManagement/dependencies/dependency').each do |n| pg = n.xpath('groupId').first.text pa = n.xpath('artifactId').first.text pv = n.xpath('version').first.text if n.xpath('version').first if g == pg and a == pa and pv result << pv end end pom_doc.xpath('/project/dependencies/dependency').each do |n| pg = n.xpath('groupId').first.text pa = n.xpath('artifactId').first.text pv = n.xpath('version').first.text if n.xpath('version').first if g == pg and a == pa and pv result << pv end end result.uniq end def coords pom_doc "#{group_id pom_doc}:#{artifact_id pom_doc}" end def upstream_coords pom_doc result = [] pom_doc.xpath('/project/parent').each do |n| groupId = n.xpath('groupId').first.text artifactId = n.xpath('artifactId').first.text result << "#{groupId}:#{artifactId}" break end pom_doc.xpath('/project/dependencies/dependency').each do |n| groupId = n.xpath('groupId').first.text artifactId = n.xpath('artifactId').first.text result << "#{groupId}:#{artifactId}" end result end def release_root? pom_doc pom_doc.xpath('/project/scm').first and pom_doc.xpath('/project/version') end def collapse_dependencies dependencies result = Hash.new dependencies.each do |coord, deps| collapsed_deps = [] deps.each { |d| collapsed_deps << d; if dependencies[d]; collapsed_deps = collapsed_deps + dependencies[d] end } result[coord] = collapsed_deps.uniq end result end def uncommitted_changes? paths result = false; paths.each do |path| if File.exists?("#{path}/.git") status = `git --git-dir #{path}/.git --work-tree #{path} status --porcelain | sed -ne '/pom.xml.releaseBackup/d;/release.properties/d;p'` if $?.exitstatus > 0 puts "#{red "Could not determine status of files in #{path}"}" exit 7 end if not status.empty? puts "#{yellow "There are uncommitted changes:"}" unless result status.split(/\n/).each { |l| s, p = l.match(/(..) (.*)/).captures; puts "#{s} #{path}/#{p}" } result = true end end end result end def commit_all paths, message paths.each do |path| if File.exists?("#{path}/.git") status = `git --git-dir #{path}/.git --work-tree #{path} status --porcelain` if $?.exitstatus > 0 puts "Could not determine status of files in #{path}" exit 7 end if not status.empty? system "git --git-dir #{path}/.git --work-tree #{path} commit -a -m \"#{message}\"" if $?.exitstatus > 0 puts "Could not commit changes in #{path}" exit 7 end status = `git --git-dir #{path}/.git --work-tree #{path} status --porcelain` if $?.exitstatus > 0 puts "Could not determine status of files in #{path}" exit 7 end if not status.empty? puts "Did not commit files in #{path}" exit 7 end status = `git --git-dir #{path}/.git --work-tree #{path} log -n 1 --pretty='%s' @{u}..` if $?.exitstatus > 0 puts "Could not determine status of files in #{path}" exit 7 end if not status.empty? status = `git --git-dir #{path}/.git --work-tree #{path} push` if $?.exitstatus > 0 puts "Could not publish changes in #{path}" exit 7 end end end end end end yolo = "" loop { case ARGV[0] when '--yolo' then ARGV.shift; yolo = "-B" else break; end } poms = Hash.new coord2path = Hash.new dependencies = Hash.new roots = [] git_roots = [] modules = ["."] modules.each do |path| pom_doc = read_pom "#{path}/pom.xml" pom_doc.xpath('/project/modules/module').each do |n| modules.include?("#{path}/#{n.text}") if modules << "#{path}/#{n.text}" end poms[path] = pom_doc coord = coords pom_doc coord2path[coord] = path dependencies[coord] = upstream_coords pom_doc # only poms that have scm section AND specify their version rather than inherit are releasable roots << coord if release_root?(pom_doc) git_roots << coord if File.exists?("#{path}/.git") end dependencies = collapse_dependencies dependencies # sort the roots into release tree order roots.sort! do |a, b| case when (dependencies[a].include?(b)) && !(dependencies[b].include?(a)) 1 when (dependencies[b].include?(a)) && !(dependencies[a].include?(b)) -1 else case dependencies[a].size <=> dependencies[b].size when -1 -1 when 1 1 when 0 a <=> b end end end exit 1 if uncommitted_changes?(git_roots.map { |e| coord2path[e] }) last_need_release = nil need_release = [] begin if not need_release.empty? exit 1 if uncommitted_changes?(git_roots.map { |e| coord2path[e] }) coord = need_release.first g, a = coord.match(/(^.*):(.*)$/i).captures path = coord2path[coord] puts "#{green "Detected changes since last release in "} #{yellow g}:#{yellow a}" system "cd #{path} && mvn release:prepare release:perform -DautoVersionSubmodules=true #{yolo}" if $?.exitstatus > 0 break end if File.exists?("#{path}/target/checkout/pom.xml") new_version = version(read_pom("#{path}/target/checkout/pom.xml")) puts "#{green "Linking downstream projects to version"} #{yellow new_version}..." dependencies.each do |d, dummy| pp = coord2path[d] if pp && dependencies[d].include?(coord) old_versions = upstream_version(poms[pp], coord) + upstream_version(read_pom("#{pp}/pom.xml"), coord) old_versions.uniq.each do |old_version| unless old_version == new_version puts " #{yellow d}" cmd = "mvn org.codehaus.mojo:versions-maven-plugin:2.1:set " + "org.codehaus.mojo:versions-maven-plugin:2.1:commit " + "-DgroupId=#{g} " + "-DartifactId=#{a} " + "-DnewVersion=#{new_version} " + "-DoldVersion=#{old_version} " + "-DprocessProject=false " + "-DupdateMatchingVersions=false" out = `#{cmd}` if $?.exitstatus > 0 puts "Could not update downstream projects:\n#{out}" exit 7 end commit_all(git_roots.map { |e| coord2path[e] }, "Updating #{g}:#{a} from #{old_version} to #{new_version}") end end end end end puts "" end last_need_release = need_release need_release = [] roots.each do |coord| path = coord2path[coord] unless `git --git-dir #{path}/.git --work-tree #{path} log -n 1 --pretty='%s'`.match(/^\[maven-release-plugin\] prepare for next development iteration/) and $?.exitstatus == 0 need_release << coord end end end while not need_release.empty? and last_need_release != need_release The second one I call `mvn-link-changed.rb` #!/usr/bin/env ruby require 'nokogiri' want_color = `git config color.ui` $color = case want_color.chomp when "true"; true when "auto"; $stdout.tty? end def red s; $color ? "\033[31m#{s}\033[0m" : s end def green s; $color ? "\033[32m#{s}\033[0m" : s end def yellow s; $color ? "\033[33m#{s}\033[0m" : s end def cyan s; $color ? "\033[36m#{s}\033[0m" : s end def grey s; $color ? "\033[1;30m#{s}\033[0m" : s end def purple s; $color ? "\033[35m#{s}\033[0m" : s end def read_pom pom_file Nokogiri::XML(File.open(pom_file)) { |config| config.nonet }.remove_namespaces! end def group_id pom_doc (pom_doc.xpath('/project/groupId').first or pom_doc.xpath('/project/parent/groupId').first).text end def artifact_id pom_doc (pom_doc.xpath('/project/artifactId').first or pom_doc.xpath('/project/parent/artifactId').first).text end def version pom_doc (pom_doc.xpath('/project/version').first or pom_doc.xpath('/project/parent/version').first).text end def upstream_version pom_doc, coord g, a = coord.match(/(^.*):(.*)$/i).captures result = [] pom_doc.xpath('/project/parent').each do |n| pg = n.xpath('groupId').first.text pa = n.xpath('artifactId').first.text pv = n.xpath('version').first.text if n.xpath('version').first if g == pg and a == pa and pv result << pv end break end pom_doc.xpath('/project/dependencyManagement/dependencies/dependency').each do |n| pg = n.xpath('groupId').first.text pa = n.xpath('artifactId').first.text pv = n.xpath('version').first.text if n.xpath('version').first if g == pg and a == pa and pv result << pv end end pom_doc.xpath('/project/dependencies/dependency').each do |n| pg = n.xpath('groupId').first.text pa = n.xpath('artifactId').first.text pv = n.xpath('version').first.text if n.xpath('version').first if g == pg and a == pa and pv result << pv end end result.uniq end def coords pom_doc "#{group_id pom_doc}:#{artifact_id pom_doc}" end def upstream_coords pom_doc result = [] pom_doc.xpath('/project/parent').each do |n| groupId = n.xpath('groupId').first.text artifactId = n.xpath('artifactId').first.text result << "#{groupId}:#{artifactId}" break end pom_doc.xpath('/project/dependencies/dependency').each do |n| groupId = n.xpath('groupId').first.text artifactId = n.xpath('artifactId').first.text result << "#{groupId}:#{artifactId}" end result end def release_root? pom_doc pom_doc.xpath('/project/scm').first and pom_doc.xpath('/project/version') end def collapse_dependencies dependencies result = Hash.new dependencies.each do |coord,deps| collapsed_deps = [] deps.each { |d| collapsed_deps << d; if dependencies[d]; collapsed_deps = collapsed_deps + dependencies[d] end } result[coord] = collapsed_deps.uniq end result end def uncommitted_changes? paths result = false; paths.each do |path| if File.exists?("#{path}/.git") status = `git --git-dir #{path}/.git --work-tree #{path} status --porcelain | sed -ne '/pom.xml.releaseBackup/d;/release.properties/d;p'` if $?.exitstatus > 0 puts "#{red "Could not determine status of files in #{path}"}" exit 7 end if not status.empty? puts "#{yellow "There are uncommitted changes:"}" unless result status.split(/\n/).each { |l| s,p = l.match(/(..) (.*)/).captures; puts "#{s} #{path}/#{p}" } result = true end end end result end def commit_all paths, message paths.each do |path| if File.exists?("#{path}/.git") status = `git --git-dir #{path}/.git --work-tree #{path} status --porcelain` if $?.exitstatus > 0 puts "Could not determine status of files in #{path}" exit 7 end if not status.empty? system "git --git-dir #{path}/.git --work-tree #{path} commit -a -m \"#{message}\"" if $?.exitstatus > 0 puts "Could not commit changes in #{path}" exit 7 end status = `git --git-dir #{path}/.git --work-tree #{path} status --porcelain` if $?.exitstatus > 0 puts "Could not determine status of files in #{path}" exit 7 end if not status.empty? puts "Did not commit files in #{path}" exit 7 end status = `git --git-dir #{path}/.git --work-tree #{path} log -n 1 --pretty='%s' @{u}..` if $?.exitstatus > 0 puts "Could not determine status of files in #{path}" exit 7 end if not status.empty? status = `git --git-dir #{path}/.git --work-tree #{path} push` if $?.exitstatus > 0 puts "Could not publish changes in #{path}" exit 7 end end end end end end poms = Hash.new coord2path = Hash.new dependencies = Hash.new roots = [] git_roots = [] modules = [ "." ] modules.each do |path| pom_doc = read_pom "#{path}/pom.xml" pom_doc.xpath('/project/modules/module').each do |n| modules.include?("#{path}/#{n.text}") if modules << "#{path}/#{n.text}" end poms[path] = pom_doc coord = coords pom_doc coord2path[coord] = path dependencies[coord] = upstream_coords pom_doc # only poms that have scm section AND specify their version rather than inherit are releasable roots << coord if release_root?(pom_doc) git_roots << coord if File.exists?("#{path}/.git") end dependencies = collapse_dependencies dependencies # sort the roots into release tree order roots.sort! do |a,b| case when (dependencies[a].include?(b)) && !(dependencies[b].include?(a)) 1 when (dependencies[b].include?(a)) && !(dependencies[a].include?(b)) -1 else case dependencies[a].size <=> dependencies[b].size when -1 -1 when 1 1 when 0 a <=> b end end end last_need_link = nil have_link = [ ] need_link = [ ] begin if not need_link.empty? coord = need_link.first g, a = coord.match(/(^.*):(.*)$/i).captures path = coord2path[coord] puts "#{green "Detected changes since last release in"} #{yellow g}:#{yellow a}" have_link << coord new_version = version(read_pom("#{path}/pom.xml")) puts "#{green "Linking downstream projects to version"} #{yellow new_version}..." dependencies.each do |d,dummy| pp = coord2path[d] if pp && dependencies[d].include?(coord) old_versions = upstream_version(poms[pp], coord) + upstream_version(read_pom("#{pp}/pom.xml"), coord) old_versions.uniq.each do |old_version| unless old_version == new_version puts " #{yellow d}" cmd = "mvn org.codehaus.mojo:versions-maven-plugin:2.1:set " + "org.codehaus.mojo:versions-maven-plugin:2.1:commit " + "-DgroupId=#{g} " + "-DartifactId=#{a} " + "-DnewVersion=#{new_version} " + "-DoldVersion=#{old_version} " + "-DprocessProject=false " + "-DupdateMatchingVersions=false" out = `#{cmd}` if $?.exitstatus > 0 puts "Could not update downstream projects:\n#{out}" exit 7 end end end end end puts "" end last_need_link = need_link need_link = [] roots.each do |coord| path = coord2path[coord] status = `git --git-dir #{path}/.git --work-tree #{path} status --porcelain` if $?.exitstatus > 0 puts "Could not determine status of files in #{path}" exit 7 end unless status.empty? and `git --git-dir #{path}/.git --work-tree #{path} log -n 1 --pretty='%s'`.match(/^\[maven-release-plugin\] prepare for next development iteration/) and $?.exitstatus == 0 unless have_link.include?(coord) need_link << coord end end end end while not need_link.empty? and last_need_link != need_link So how do you use these two scripts? Well what I tend to do is have a local aggregator project when I am working on different components. I check out all the components and the local aggregator project (typically not checked in to SCM) has <module> entries for each of the different libraries. Now normally I'm working away using release versions of the libraries... but oh look the foobar library has a bug in it. I go and fix the foobar module... then I run $ mvn-link-changed.rb And presto, everything gets updated to use the -SNAPSHOT dependencies from the local aggregator root... Now the script is smart enough to only link in modules that have changed since the last use of the maven release plugin, so most things will stay on the release version until you update the code. When it comes time to push releases $ mvn-release-all.rb --yolo (You can omit the --yolo option if you want to be prompted by the release plugin for each SCM root) And that will release, in dependency order, all the modules that have changes since the last release. I should probably write this up as a blog post... but my scripts are not yet as robust as I'd like... they work for me... largely because I have everything in git On 13 June 2014 09:41, Hohl, Gerrit <g.h...@aurenz.de> wrote: > Hello everyone, :) > > wow, I got a lot of useful responses. Thanks a lot, Michael, Ron, Barrie > and Pascal. > And sorry for the double mail. Seems the mailing list got it twice. > > We already have a Nexus running here (v2.6.2-01 - I know: There is > v2.8.1-01 available) and use it as a mirror / proxy as well as for > uploading some JAR and POM files which are not available in the public > repositories. > > Instead of the Maven Versions Plugin we came across the Maven Release > Plugin. This way you can check in all the time, but only once the build > process creates a stable release by modifying the pom.xml files, building, > checking in this version of the pom.xml into the SCM, tagging or branching > this version in the SCM, modifying the pom.xml again (back to snaphots) and > checking it in again. > But we asked ourselves if this is really the best solution as it seemed a > little bit complex and if there isn't a more simple solution for doing that. > I have to admit that a colleague was working on that issue and not me. I > will discuss it with him again. > > Another colleague and I said the same like Ron wrote. That internal > libraries should be handled like external ones. > Our problem at the moment is that the libraries as well as the application > change every day. We don't have separated teams for applications and > libraries - we are too small for that. On the other side other developers > of the team have to rely on changes which are made. So we have to publish > them to the SCM. Here we face the problem that we also have to deploy them > to Nexus as the other party maybe only works on one project, but doesn't > have the library checked out from the SCM at the same time. So if project A > is checked in you also have to deploy library B. Otherwise the other > developer which checks out project A won't be able to start it, maybe even > to compile it, because project A in the SCM depends on a newer library B > version than the developer has in his local repository. > My argument in the discussion back then was that this should be solved by > snapshots. Especially if you have a build server which is triggered by > changes in the SCM you would be able to have the newest version of the > library always available in your Nexus repository. > > We read many articles on the Internet so far. And we read the following > books: > > Maven 2 - Eine Einführung, aktuell zur Version 2.0.9 (Maven 2 - An > introduction) > Author: Kai Uwe Bachmann > Publisher: Addision-Wesley > ISBN: 978-3-8273-2835-9 > It is a really only an introduction and many things are not mentioned or > discusses in that book. Kai talks only about very simply examples. He > doesn't write about more complex problems you will face for sure if you > work a little bit longer and maybe on bigger projects using Maven. > > Jenkins - kurz & gut > Author: Mario Behrendt > Publisher: O'Reilly > ISBN: 978-3-86899-127-7 > This books is not directly related to Maven, but also mentions it. It is a > short overview of CI using Jenkins. I guess you won't get Jenkins running > if you only read this book. But maybe it is a good help for making a > decision if you want to use Jenkins or not. > > Jenkins - The Definitive Guide > Author: John Ferguson Smart > Publisher: O'Reilly > ISBN: 978-1-449-30535-2 > Also not directly related to Maven, but explain how to use Jenkins and > Maven together. > > I also bought recently: > Continuous Delivery: Reliable Software Releases Through Build, Test, and > Deployment Automation > Authors: Jez Humble, David Farley > Publisher: Addision-Wesley > ISBN: 978-0321601919 > I haven't read it yet. But I hope to get a deeper understanding of the > whole process. But I'm not sure if it will help me solving our problems. > > I had a look on the books on the Maven site. Most look like they are > already old and dealing with Maven 2 (like the book we have - unfortunately > - also does). And the newest one got some poor reviews (okay, they are > subjective, so maybe the book isn't that bad). Maybe one of you can give a > recommendation for a book which is not dedicated to beginners, but instead > handles some issues of developers who are already on the next level (but > not on the highest level yet). > > Of course we are also reading the online documentation. But I have to > admit that it sometimes lacks the one or another information which you find > in some blogs or articles after some search. And some of the 3rd party > plugins are explained really poorly (okay, that is not a problem of the > Maven project). That doesn't make it easier as Maven offers some many > possibilities and therefore *can* get complex. > > We also found a presentation on the Internet which described a solution > like Pascal mentioned in his mail. But doesn't that mean that you will have > hundreds and hundreds of releases in your repository? And doesn't it mean > you have to update the pom.xml files of your projects which are developed > simultaneously all the time? Somehow I prefer the idea - like Ron - of > handling internal libraries not different from external libraries. But I > also see the points of people who argument for the CD way-of-life. > Nevertheless the links Pascal provided are very interesting. And I found > even a few posts of Jez Humble, the author of that CD book I haven't read > yet. :D I will definitely read the forum discussion and also have a look at > the YouTube video. > > Again I want to thank all to all the people who responded to my mail. Now > I have some material to read through and discuss it with my colleagues. I > hope you're not getting mad if I come back for more questions. :) > > Regards, > Gerrit > > > -----Ursprüngliche Nachricht----- > Von: Pascal Rapicault [mailto:pas...@rapicault.net] > Gesendet: Freitag, 13. Juni 2014 04:32 > An: Maven Users List > Betreff: Re: Complex Maven projects - Tutorials? Books? > > If you are really aiming at doing continuous delivery (any potential build > can be pushed to prod), then SNAPSHOT is not a great way to deal with > dependencies since you will not be able to exactly know what you ship. To > avoid this, one practice is to use the build number in the artifact version > (1.0.0-b1 or 1.0.1). > This has of course had the drawback that now you have to update the > pom.xml of components using a specific artifact (move from build 1 to 2) > but this also gives you greater control on the rate at which you consume > libraries. > You may be interested in these articles: > - > > http://maven.40175.n5.nabble.com/Continuous-Delivery-and-Maven-td3245370.html > - > > http://stackoverflow.com/questions/18456111/what-is-the-maven-way-for-project-versions-when-doing-continuous-delivery > > That said, if you add Artifactory to the mix, you can leverage its > capabilities of obtaining specific versions of a SNAPSHOT through matrix > parameters > ( > https://www.jfrog.com/confluence/display/RTF/Using+Properties+in+Deployment+and+Resolution > ) > quite handy. One example where this comes handy is when you split your > build process over multiple jenkins jobs and you want to make sure that you > use the same artifact throughout the process and this w/o blocking the > whole "pipeline" for the whole duration of the process. > > HTH > > Pascal > On 12/06/2014 10:46 AM, Hohl, Gerrit wrote: > > Hello everyone, :) > > > > > > > > I have a question which is not about a specific problem with Maven, > > but more a general question. > > > > I hope it is okay to ask this question here. > > > > > > > > We use Maven and Jenkins for about 1.5 years now, I guess. Until now > > the Maven projects have been very simple and - let's say - very > monolithic. > > > > But recently we identify more and more internal libraries in our > > products. Of course we don't want to share this libraries by > > copy-n-paste between the products - especially as we have Maven. > > > > So we started to read books, tutorials on the Internet and so on. But > > most of them only deal with simple projects. They don't cover e.g. > > versioning the build process (especially if your build process > > consists of more than just one step). They also don't cover the > > problems of developing the libraries while your developing the > > products which depend on them. Especially at the beginning your > > libraries will go through a lot of changes. A few name snapshots as a > > solution, but don't explain how you can work using them, how you can > > use them in your pom.xml and how you deal with them if you finally > > switch your product and/or your library from the snapshot state to the > > release state. A few also say that you shouldn't use snapshots at all > > because it will result in many problems (e.g. having -SNAPSHOT entries > > in your pom.xml). Nightly builds or build triggered by the SCM are also > an issue here. > > > > > > > > Does someone know a good book or tutorial which handles all of these > > issues around Maven and CI/CD in more depth? > > > > > > > > > > > > Regards, > > > > Gerrit > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > For additional commands, e-mail: users-h...@maven.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > For additional commands, e-mail: users-h...@maven.apache.org > >