Ah hah! Patrick figured it out: https://github.com/apache/incubator-spark/commit/905edf59db662868f55525118131cf102d366587. The buildnumber plugin and Debian packaging works correctly again. Thanks! I'll go through the poms tomorrow and see if there are any other such incompletely overriden elements that could be causing us troubles.
Right now I am seeing two problems with the maven build: 1) I just had a build fail from exhausted PermGen, so we probably need to increase MaxPermGen in spark/pom.xml; 2) Not all of the modules got picked up in the recent invocations of the maven-release-plugin -- did your prepare & release use -Phadoop2-yar,repl-bin , Patrick? On Wed, Sep 11, 2013 at 10:31 PM, Mark Hamstra <[email protected]>wrote: > Yes, exactly. If I comment out the reference to the parent Apache pom, > then the buildnumber plugin works correctly. Similarly if I leave the > parent Apache pom reference in place but go into my local .m2 cache and > comment out just the <scm> in the org.apache:apache pom. In other words, > if both <scm>s are present, the buildnumber plugin only sees the SVN one > from the Apache pom, which makes things go wrong. > > > On Wed, Sep 11, 2013 at 9:18 PM, Henry Saputra <[email protected]>wrote: > >> Hi Mark, >> >> But the spark main pom.xml contains <scm> element that should override >> <scm> from the parent pom.xml. >> >> Are you saying the pom.xml from parent Apache pom.xml overrides the >> Spark <scm> definition instead? >> >> >> >> >> On Wed, Sep 11, 2013 at 3:04 PM, Mark Hamstra <[email protected]> >> wrote: >> > I'm in the process of trying to build Apache-fied Spark into our stack >> at >> > ClearStory. We've been using Maven to do Debian packaging as found in >> the >> > repl-bin module for quite a while, but that doesn't work now. To see >> the >> > failure, you need to build repl-bin with the `deb` profile -- e.g. "mvn >> > -Prepl-bin,deb install" from the project directory. The problem is with >> > the buildnumber-maven-plugin, which is used to attach the 8-digit short >> SHA >> > of the current git commit to the Debian package -- so, e.g., >> > spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber plugin relies >> upon >> > the <scm> tag to determine which source code management system should be >> > used to lookup the version number. >> > >> > That all worked great until the following was added to spark/pom.xml: >> > >> > <parent> >> > <groupId>org.apache</groupId> >> > <artifactId>apache</artifactId> >> > <version>11</version> >> > </parent> >> > >> > The problem is that the parent pom that the above includes has this >> within >> > it: >> > >> > <scm> >> > <connection>scm:svn: >> > http://svn.apache.org/repos/asf/maven/pom/tags/apache-11</connection >> > <developerConnection>scm:svn: >> > https://svn.apache.org/repos/asf/maven/pom/tags/apache-11 >> > </developerConnection> >> > <url>http://svn.apache.org/viewvc/maven/pom/tags/apache-11</url> >> > </scm> >> > >> > And that essentially overrides the <scm> section of spark/pom.xml, which >> > means that the buildnumber plugin thinks that we are using SVN instead >> of >> > git, and the SHA lookup fails. >> > >> > So, that is one known problem in the Maven build. I'm not at all >> certain >> > that there aren't others or lurking problems because of different >> versions >> > of dependencies, plugins, etc. being included by that org.apache:apache >> > artifact and also specified within the Spark pom files. >> > >> > I'm pretty sure that we can't not include the equivalent of >> > org.apache:apache:11, since there is a lot of important stuff in there >> to >> > do Apache release management, etc. So, is there another, equivalent >> > artifact that we can use that doesn't confuse the <scm> issue, or some >> > other way around this problem? It looks like there are some other >> > git-specific plugins out there that we may be able to use instead of >> > buildnumber-maven-plugin, so that may be a valid and necessary solution >> to >> > the Debian packaging problem. >> > >> > Can someone who knows Maven better than I take a look at the overlap >> > between org.apache:apache and the Spark pom files to see if there are >> > problems lurking there? Even if there aren't problems other than with >> the >> > buildnumber plugin, we may be able to trim a lot out of the Spark poms >> that >> > is already present in the apache parent pom. >> > >
