Re: Maven SCM trouble
It looks like scm in the pom is only used to upload artifacts to Maven Central and isn't needed for building. Explained here: http://books.sonatype.com/nexus-book/reference/_introduction_7.html If you run into a transitive dependency problem, there is a nice way to exclude them. Explained here: http://books.sonatype.com/mvnref-book/reference/pom-relationships-sect-project-dependencies.html#pom-relationships-sect-conflict By far the best Maven documentation is in Sonatype's free books. They're accurate and up to date, unlike the Apache Maven docs or StackOverflow. http://www.sonatype.com/resources/books Gary Struthers On Sep 11, 2013, at 11:21 PM, Mark Hamstra m...@clearstorydata.com wrote: Ah hah! Patrick figured it out: https://github.com/apache/incubator-spark/commit/905edf59db662868f55525118131cf102d366587. The buildnumber plugin and Debian packaging works correctly again. Thanks! I'll go through the poms tomorrow and see if there are any other such incompletely overriden elements that could be causing us troubles. Right now I am seeing two problems with the maven build: 1) I just had a build fail from exhausted PermGen, so we probably need to increase MaxPermGen in spark/pom.xml; 2) Not all of the modules got picked up in the recent invocations of the maven-release-plugin -- did your prepare release use -Phadoop2-yar,repl-bin , Patrick? On Wed, Sep 11, 2013 at 10:31 PM, Mark Hamstra m...@clearstorydata.comwrote: Yes, exactly. If I comment out the reference to the parent Apache pom, then the buildnumber plugin works correctly. Similarly if I leave the parent Apache pom reference in place but go into my local .m2 cache and comment out just the scm in the org.apache:apache pom. In other words, if both scms are present, the buildnumber plugin only sees the SVN one from the Apache pom, which makes things go wrong. On Wed, Sep 11, 2013 at 9:18 PM, Henry Saputra henry.sapu...@gmail.comwrote: Hi Mark, But the spark main pom.xml contains scm element that should override scm from the parent pom.xml. Are you saying the pom.xml from parent Apache pom.xml overrides the Spark scm definition instead? On Wed, Sep 11, 2013 at 3:04 PM, Mark Hamstra m...@clearstorydata.com wrote: I'm in the process of trying to build Apache-fied Spark into our stack at ClearStory. We've been using Maven to do Debian packaging as found in the repl-bin module for quite a while, but that doesn't work now. To see the failure, you need to build repl-bin with the `deb` profile -- e.g. mvn -Prepl-bin,deb install from the project directory. The problem is with the buildnumber-maven-plugin, which is used to attach the 8-digit short SHA of the current git commit to the Debian package -- so, e.g., spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber plugin relies upon the scm tag to determine which source code management system should be used to lookup the version number. That all worked great until the following was added to spark/pom.xml: parent groupIdorg.apache/groupId artifactIdapache/artifactId version11/version /parent The problem is that the parent pom that the above includes has this within it: scm connectionscm:svn: http://svn.apache.org/repos/asf/maven/pom/tags/apache-11/connection developerConnectionscm:svn: https://svn.apache.org/repos/asf/maven/pom/tags/apache-11 /developerConnection urlhttp://svn.apache.org/viewvc/maven/pom/tags/apache-11/url /scm And that essentially overrides the scm section of spark/pom.xml, which means that the buildnumber plugin thinks that we are using SVN instead of git, and the SHA lookup fails. So, that is one known problem in the Maven build. I'm not at all certain that there aren't others or lurking problems because of different versions of dependencies, plugins, etc. being included by that org.apache:apache artifact and also specified within the Spark pom files. I'm pretty sure that we can't not include the equivalent of org.apache:apache:11, since there is a lot of important stuff in there to do Apache release management, etc. So, is there another, equivalent artifact that we can use that doesn't confuse the scm issue, or some other way around this problem? It looks like there are some other git-specific plugins out there that we may be able to use instead of buildnumber-maven-plugin, so that may be a valid and necessary solution to the Debian packaging problem. Can someone who knows Maven better than I take a look at the overlap between org.apache:apache and the Spark pom files to see if there are problems lurking there? Even if there aren't problems other than with the buildnumber plugin, we may be able to trim a lot out of the Spark poms that is already present in the apache parent pom.
Re: Maven SCM trouble
I was about to suggest that. Thanks Patrick. - Henry On Wednesday, September 11, 2013, Patrick Wendell wrote: Ya I amended this for pushing the releases to maven. I actually changed that later to a different URI because it was incorrect (should have been https://) On Wed, Sep 11, 2013 at 11:41 PM, Gary Struthers agilej...@earthlink.net wrote: It looks like scm in the pom is only used to upload artifacts to Maven Central and isn't needed for building. Explained here: http://books.sonatype.com/nexus-book/reference/_introduction_7.html If you run into a transitive dependency problem, there is a nice way to exclude them. Explained here: http://books.sonatype.com/mvnref-book/reference/pom-relationships-sect-project-dependencies.html#pom-relationships-sect-conflict By far the best Maven documentation is in Sonatype's free books. They're accurate and up to date, unlike the Apache Maven docs or StackOverflow. http://www.sonatype.com/resources/books Gary Struthers On Sep 11, 2013, at 11:21 PM, Mark Hamstra m...@clearstorydata.com wrote: Ah hah! Patrick figured it out: https://github.com/apache/incubator-spark/commit/905edf59db662868f55525118131cf102d366587 . The buildnumber plugin and Debian packaging works correctly again. Thanks! I'll go through the poms tomorrow and see if there are any other such incompletely overriden elements that could be causing us troubles. Right now I am seeing two problems with the maven build: 1) I just had a build fail from exhausted PermGen, so we probably need to increase MaxPermGen in spark/pom.xml; 2) Not all of the modules got picked up in the recent invocations of the maven-release-plugin -- did your prepare release use -Phadoop2-yar,repl-bin , Patrick? On Wed, Sep 11, 2013 at 10:31 PM, Mark Hamstra m...@clearstorydata.com wrote: Yes, exactly. If I comment out the reference to the parent Apache pom, then the buildnumber plugin works correctly. Similarly if I leave the parent Apache pom reference in place but go into my local .m2 cache and comment out just the scm in the org.apache:apache pom. In other words, if both scms are present, the buildnumber plugin only sees the SVN one from the Apache pom, which makes things go wrong. On Wed, Sep 11, 2013 at 9:18 PM, Henry Saputra henry.sapu...@gmail.comwrote: Hi Mark, But the spark main pom.xml contains scm element that should override scm from the parent pom.xml. Are you saying the pom.xml from parent Apache pom.xml overrides the Spark scm definition instead? On Wed, Sep 11, 2013 at 3:04 PM, Mark Hamstra m...@clearstorydata.com wrote: I'm in the process of trying to build Apache-fied Spark into our stack at ClearStory. We've been using Maven to do Debian packaging as found in the repl-bin module for quite a while, but that doesn't work now. To see the failure, you need to build repl-bin with the `deb` profile -- e.g. mvn -Prepl-bin,deb install from the project directory. The problem is with the buildnumber-maven-plugin, which is used to attach the 8-digit short SHA of the current git commit to the Debian package -- so, e.g., spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber pl
Re: [VOTE] Release Apache Spark 0.8.0-incubating (RC3)
Hey guys, we actually decided on a slightly different naming convention for the downloads. I'm going to amend the files in the next few minutes... in case anyone happens to be looking *this instant* (which I doubt) hold off until I update them. On Thu, Sep 12, 2013 at 3:57 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark (incubating) version 0.8.0. This will be the first incubator release for Spark in Apache. The tag to be voted on is v0.8.0-incubating (commit ffacd17): https://github.com/apache/incubator-spark/releases/tag/v0.8.0-incubating The release files, including signatures, digests, etc can be found at: http://people.apache.org/~pwendell/spark-0.8.0-incubating-rc3/files/ The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-034/org/apache/spark/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-0.8.0-incubating-rc3/docs/ Please vote on releasing this package as Apache Spark 0.8.0-incubating! The vote is open until Saturday, June 13th at 23:00 UTC and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Spark 0.8.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.incubator.apache.org/
Re: [VOTE] Release Apache Spark 0.8.0-incubating (RC3)
Fixed! On Thu, Sep 12, 2013 at 4:22 PM, Patrick Wendell pwend...@gmail.com wrote: Hey guys, we actually decided on a slightly different naming convention for the downloads. I'm going to amend the files in the next few minutes... in case anyone happens to be looking *this instant* (which I doubt) hold off until I update them. On Thu, Sep 12, 2013 at 3:57 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark (incubating) version 0.8.0. This will be the first incubator release for Spark in Apache. The tag to be voted on is v0.8.0-incubating (commit ffacd17): https://github.com/apache/incubator-spark/releases/tag/v0.8.0-incubating The release files, including signatures, digests, etc can be found at: http://people.apache.org/~pwendell/spark-0.8.0-incubating-rc3/files/ The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-034/org/apache/spark/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-0.8.0-incubating-rc3/docs/ Please vote on releasing this package as Apache Spark 0.8.0-incubating! The vote is open until Saturday, June 13th at 23:00 UTC and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Spark 0.8.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.incubator.apache.org/
[VOTE] Release Apache Spark 0.8.0-incubating (RC3)
Please vote on releasing the following candidate as Apache Spark (incubating) version 0.8.0. This will be the first incubator release for Spark in Apache. The tag to be voted on is v0.8.0-incubating (commit ffacd17): https://github.com/apache/incubator-spark/releases/tag/v0.8.0-incubating The release files, including signatures, digests, etc can be found at: http://people.apache.org/~pwendell/spark-0.8.0-incubating-rc3/files/ The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-034/org/apache/spark/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-0.8.0-incubating-rc3/docs/ Please vote on releasing this package as Apache Spark 0.8.0-incubating! The vote is open until Saturday, June 13th at 23:00 UTC and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Spark 0.8.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.incubator.apache.org/
