It looks like scm in the pom is only used to upload artifacts to Maven Central and isn't needed for building. Explained here: http://books.sonatype.com/nexus-book/reference/_introduction_7.html
If you run into a transitive dependency problem, there is a nice way to exclude them. Explained here: http://books.sonatype.com/mvnref-book/reference/pom-relationships-sect-project-dependencies.html#pom-relationships-sect-conflict By far the best Maven documentation is in Sonatype's free books. They're accurate and up to date, unlike the Apache Maven docs or StackOverflow. http://www.sonatype.com/resources/books Gary Struthers On Sep 11, 2013, at 11:21 PM, Mark Hamstra <m...@clearstorydata.com> wrote: > Ah hah! Patrick figured it out: > https://github.com/apache/incubator-spark/commit/905edf59db662868f55525118131cf102d366587. > The buildnumber plugin and Debian packaging works correctly again. > Thanks! I'll go through the poms tomorrow and see if there are any other > such incompletely overriden elements that could be causing us troubles. > > Right now I am seeing two problems with the maven build: 1) I just had a > build fail from exhausted PermGen, so we probably need to increase > MaxPermGen in spark/pom.xml; 2) Not all of the modules got picked up in the > recent invocations of the maven-release-plugin -- did your prepare & > release use -Phadoop2-yar,repl-bin , Patrick? > > > > On Wed, Sep 11, 2013 at 10:31 PM, Mark Hamstra <m...@clearstorydata.com>wrote: > >> Yes, exactly. If I comment out the reference to the parent Apache pom, >> then the buildnumber plugin works correctly. Similarly if I leave the >> parent Apache pom reference in place but go into my local .m2 cache and >> comment out just the <scm> in the org.apache:apache pom. In other words, >> if both <scm>s are present, the buildnumber plugin only sees the SVN one >> from the Apache pom, which makes things go wrong. >> >> >> On Wed, Sep 11, 2013 at 9:18 PM, Henry Saputra >> <henry.sapu...@gmail.com>wrote: >> >>> Hi Mark, >>> >>> But the spark main pom.xml contains <scm> element that should override >>> <scm> from the parent pom.xml. >>> >>> Are you saying the pom.xml from parent Apache pom.xml overrides the >>> Spark <scm> definition instead? >>> >>> >>> >>> >>> On Wed, Sep 11, 2013 at 3:04 PM, Mark Hamstra <m...@clearstorydata.com> >>> wrote: >>>> I'm in the process of trying to build Apache-fied Spark into our stack >>> at >>>> ClearStory. We've been using Maven to do Debian packaging as found in >>> the >>>> repl-bin module for quite a while, but that doesn't work now. To see >>> the >>>> failure, you need to build repl-bin with the `deb` profile -- e.g. "mvn >>>> -Prepl-bin,deb install" from the project directory. The problem is with >>>> the buildnumber-maven-plugin, which is used to attach the 8-digit short >>> SHA >>>> of the current git commit to the Debian package -- so, e.g., >>>> spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber plugin relies >>> upon >>>> the <scm> tag to determine which source code management system should be >>>> used to lookup the version number. >>>> >>>> That all worked great until the following was added to spark/pom.xml: >>>> >>>> <parent> >>>> <groupId>org.apache</groupId> >>>> <artifactId>apache</artifactId> >>>> <version>11</version> >>>> </parent> >>>> >>>> The problem is that the parent pom that the above includes has this >>> within >>>> it: >>>> >>>> <scm> >>>> <connection>scm:svn: >>>> http://svn.apache.org/repos/asf/maven/pom/tags/apache-11</connection >>>> <developerConnection>scm:svn: >>>> https://svn.apache.org/repos/asf/maven/pom/tags/apache-11 >>>> </developerConnection> >>>> <url>http://svn.apache.org/viewvc/maven/pom/tags/apache-11</url> >>>> </scm> >>>> >>>> And that essentially overrides the <scm> section of spark/pom.xml, which >>>> means that the buildnumber plugin thinks that we are using SVN instead >>> of >>>> git, and the SHA lookup fails. >>>> >>>> So, that is one known problem in the Maven build. I'm not at all >>> certain >>>> that there aren't others or lurking problems because of different >>> versions >>>> of dependencies, plugins, etc. being included by that org.apache:apache >>>> artifact and also specified within the Spark pom files. >>>> >>>> I'm pretty sure that we can't not include the equivalent of >>>> org.apache:apache:11, since there is a lot of important stuff in there >>> to >>>> do Apache release management, etc. So, is there another, equivalent >>>> artifact that we can use that doesn't confuse the <scm> issue, or some >>>> other way around this problem? It looks like there are some other >>>> git-specific plugins out there that we may be able to use instead of >>>> buildnumber-maven-plugin, so that may be a valid and necessary solution >>> to >>>> the Debian packaging problem. >>>> >>>> Can someone who knows Maven better than I take a look at the overlap >>>> between org.apache:apache and the Spark pom files to see if there are >>>> problems lurking there? Even if there aren't problems other than with >>> the >>>> buildnumber plugin, we may be able to trim a lot out of the Spark poms >>> that >>>> is already present in the apache parent pom. >>> >> >>
