I don't know how to make sense of the numbers, but here's what I've got from a very small sample size.
For both v0.8.0-incubating and v0.8.1-incubating, building separate assemblies is faster than `./sbt/sbt assembly` and the times for building separate assemblies for 0.8.0 and 0.8.1 are about the same. For v0.8.0-incubating, `./sbt/sbt assembly` takes about 2.5x as long as the sum of the separate assemblies. For v0.8.1-incubating, `./sbt/sbt assembly` takes almost 8x as long as the sum of the separate assemblies. Weird. On Wed, Dec 11, 2013 at 11:49 AM, Patrick Wendell <pwend...@gmail.com>wrote: > I'll +1 myself also. > > For anyone who has the slow build problem: does this issue happen when > building v0.8.0-incubating also? Trying to figure out whether it's > related to something we added in 0.8.1 or if it's a long standing > issue. > > - Patrick > > On Wed, Dec 11, 2013 at 10:39 AM, Matei Zaharia <matei.zaha...@gmail.com> > wrote: > > Woah, weird, but definitely good to know. > > > > If you’re doing Spark development, there’s also a more convenient option > added by Shivaram in the master branch. You can do sbt assemble-deps to > package *just* the dependencies of each project in a special assembly JAR, > and then use sbt compile to update the code. This will use the classes > directly out of the target/scala-2.9.3/classes directories. You have to > redo assemble-deps only if your external dependencies change. > > > > Matei > > > > On Dec 11, 2013, at 1:04 AM, Prashant Sharma <scrapco...@gmail.com> > wrote: > > > >> I hope this PR https://github.com/apache/incubator-spark/pull/252 can > help. > >> Again this is not a blocker for the release from my side either. > >> > >> > >> On Wed, Dec 11, 2013 at 2:14 PM, Mark Hamstra <m...@clearstorydata.com > >wrote: > >> > >>> Interesting, and confirmed: On my machine where `./sbt/sbt assembly` > takes > >>> a long, long, looooong time to complete (a MBP, in my case), building > three > >>> separate assemblies (`./sbt/sbt assembly/assembly`, `./sbt/sbt > >>> examples/assembly`, `./sbt/sbt tools/assembly`) takes much, much less > time. > >>> > >>> > >>> > >>> On Wed, Dec 11, 2013 at 12:02 AM, Prashant Sharma < > scrapco...@gmail.com > >>>> wrote: > >>> > >>>> forgot to mention, after running sbt/sbt assembly/assembly running > >>> sbt/sbt > >>>> examples/assembly takes just 37s. Not to mention my hardware is not > >>> really > >>>> great. > >>>> > >>>> > >>>> On Wed, Dec 11, 2013 at 1:28 PM, Prashant Sharma < > scrapco...@gmail.com > >>>>> wrote: > >>>> > >>>>> Hi Patrick and Matei, > >>>>> > >>>>> Was trying out this and followed the quick start guide which says do > >>>>> sbt/sbt assembly, like few others I was also stuck for few minutes on > >>>>> linux. On the other hand if I use sbt/sbt assembly/assembly it is > much > >>>>> faster. > >>>>> > >>>>> Should we change the documentation to reflect this. It will not be > >>> great > >>>>> for first time users to get stuck there. > >>>>> > >>>>> > >>>>> On Wed, Dec 11, 2013 at 9:54 AM, Matei Zaharia < > >>> matei.zaha...@gmail.com > >>>>> wrote: > >>>>> > >>>>>> +1 > >>>>>> > >>>>>> Built and tested it on Mac OS X. > >>>>>> > >>>>>> Matei > >>>>>> > >>>>>> > >>>>>> On Dec 10, 2013, at 4:49 PM, Patrick Wendell <pwend...@gmail.com> > >>>> wrote: > >>>>>> > >>>>>>> Please vote on releasing the following candidate as Apache Spark > >>>>>>> (incubating) version 0.8.1. > >>>>>>> > >>>>>>> The tag to be voted on is v0.8.1-incubating (commit b87d31d): > >>>>>>> > >>>>>> > >>>> > >>> > https://git-wip-us.apache.org/repos/asf/incubator-spark/repo?p=incubator-spark.git;a=commit;h=b87d31dd8eb4b4e47c0138e9242d0dd6922c8c4e > >>>>>>> > >>>>>>> The release files, including signatures, digests, etc can be found > >>> at: > >>>>>>> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc4/ > >>>>>>> > >>>>>>> Release artifacts are signed with the following key: > >>>>>>> https://people.apache.org/keys/committer/pwendell.asc > >>>>>>> > >>>>>>> The staging repository for this release can be found at: > >>>>>>> > >>>> > https://repository.apache.org/content/repositories/orgapachespark-040/ > >>>>>>> > >>>>>>> The documentation corresponding to this release can be found at: > >>>>>>> > http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc4-docs/ > >>>>>>> > >>>>>>> For information about the contents of this release see: > >>>>>>> > >>>>>> > >>>> > >>> > https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=blob;f=CHANGES.txt;h=ce0aeab524505b63c7999e0371157ac2def6fe1c;hb=branch-0.8 > >>>>>>> > >>>>>>> Please vote on releasing this package as Apache Spark > >>>> 0.8.1-incubating! > >>>>>>> > >>>>>>> The vote is open until Saturday, December 14th at 01:00 UTC and > >>>>>>> passes if a majority of at least 3 +1 PPMC votes are cast. > >>>>>>> > >>>>>>> [ ] +1 Release this package as Apache Spark 0.8.1-incubating > >>>>>>> [ ] -1 Do not release this package because ... > >>>>>>> > >>>>>>> To learn more about Apache Spark, please see > >>>>>>> http://spark.incubator.apache.org/ > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> s > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> s > >>>> > >>> > >> > >> > >> > >> -- > >> s > > >
