I'd be personally fine with a standard workflow of assemble-deps + packaging just the Spark files as separate packages, if it speeds up everyone's development time.
On Wed, Dec 11, 2013 at 1:10 PM, Mark Hamstra <m...@clearstorydata.com>wrote: > I don't know how to make sense of the numbers, but here's what I've got > from a very small sample size. > > For both v0.8.0-incubating and v0.8.1-incubating, building separate > assemblies is faster than `./sbt/sbt assembly` and the times for building > separate assemblies for 0.8.0 and 0.8.1 are about the same. > > For v0.8.0-incubating, `./sbt/sbt assembly` takes about 2.5x as long as the > sum of the separate assemblies. > For v0.8.1-incubating, `./sbt/sbt assembly` takes almost 8x as long as the > sum of the separate assemblies. > > Weird. > > > On Wed, Dec 11, 2013 at 11:49 AM, Patrick Wendell <pwend...@gmail.com > >wrote: > > > I'll +1 myself also. > > > > For anyone who has the slow build problem: does this issue happen when > > building v0.8.0-incubating also? Trying to figure out whether it's > > related to something we added in 0.8.1 or if it's a long standing > > issue. > > > > - Patrick > > > > On Wed, Dec 11, 2013 at 10:39 AM, Matei Zaharia <matei.zaha...@gmail.com > > > > wrote: > > > Woah, weird, but definitely good to know. > > > > > > If you’re doing Spark development, there’s also a more convenient > option > > added by Shivaram in the master branch. You can do sbt assemble-deps to > > package *just* the dependencies of each project in a special assembly > JAR, > > and then use sbt compile to update the code. This will use the classes > > directly out of the target/scala-2.9.3/classes directories. You have to > > redo assemble-deps only if your external dependencies change. > > > > > > Matei > > > > > > On Dec 11, 2013, at 1:04 AM, Prashant Sharma <scrapco...@gmail.com> > > wrote: > > > > > >> I hope this PR https://github.com/apache/incubator-spark/pull/252 can > > help. > > >> Again this is not a blocker for the release from my side either. > > >> > > >> > > >> On Wed, Dec 11, 2013 at 2:14 PM, Mark Hamstra < > m...@clearstorydata.com > > >wrote: > > >> > > >>> Interesting, and confirmed: On my machine where `./sbt/sbt assembly` > > takes > > >>> a long, long, looooong time to complete (a MBP, in my case), building > > three > > >>> separate assemblies (`./sbt/sbt assembly/assembly`, `./sbt/sbt > > >>> examples/assembly`, `./sbt/sbt tools/assembly`) takes much, much less > > time. > > >>> > > >>> > > >>> > > >>> On Wed, Dec 11, 2013 at 12:02 AM, Prashant Sharma < > > scrapco...@gmail.com > > >>>> wrote: > > >>> > > >>>> forgot to mention, after running sbt/sbt assembly/assembly running > > >>> sbt/sbt > > >>>> examples/assembly takes just 37s. Not to mention my hardware is not > > >>> really > > >>>> great. > > >>>> > > >>>> > > >>>> On Wed, Dec 11, 2013 at 1:28 PM, Prashant Sharma < > > scrapco...@gmail.com > > >>>>> wrote: > > >>>> > > >>>>> Hi Patrick and Matei, > > >>>>> > > >>>>> Was trying out this and followed the quick start guide which says > do > > >>>>> sbt/sbt assembly, like few others I was also stuck for few minutes > on > > >>>>> linux. On the other hand if I use sbt/sbt assembly/assembly it is > > much > > >>>>> faster. > > >>>>> > > >>>>> Should we change the documentation to reflect this. It will not be > > >>> great > > >>>>> for first time users to get stuck there. > > >>>>> > > >>>>> > > >>>>> On Wed, Dec 11, 2013 at 9:54 AM, Matei Zaharia < > > >>> matei.zaha...@gmail.com > > >>>>> wrote: > > >>>>> > > >>>>>> +1 > > >>>>>> > > >>>>>> Built and tested it on Mac OS X. > > >>>>>> > > >>>>>> Matei > > >>>>>> > > >>>>>> > > >>>>>> On Dec 10, 2013, at 4:49 PM, Patrick Wendell <pwend...@gmail.com> > > >>>> wrote: > > >>>>>> > > >>>>>>> Please vote on releasing the following candidate as Apache Spark > > >>>>>>> (incubating) version 0.8.1. > > >>>>>>> > > >>>>>>> The tag to be voted on is v0.8.1-incubating (commit b87d31d): > > >>>>>>> > > >>>>>> > > >>>> > > >>> > > > https://git-wip-us.apache.org/repos/asf/incubator-spark/repo?p=incubator-spark.git;a=commit;h=b87d31dd8eb4b4e47c0138e9242d0dd6922c8c4e > > >>>>>>> > > >>>>>>> The release files, including signatures, digests, etc can be > found > > >>> at: > > >>>>>>> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc4/ > > >>>>>>> > > >>>>>>> Release artifacts are signed with the following key: > > >>>>>>> https://people.apache.org/keys/committer/pwendell.asc > > >>>>>>> > > >>>>>>> The staging repository for this release can be found at: > > >>>>>>> > > >>>> > > https://repository.apache.org/content/repositories/orgapachespark-040/ > > >>>>>>> > > >>>>>>> The documentation corresponding to this release can be found at: > > >>>>>>> > > http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc4-docs/ > > >>>>>>> > > >>>>>>> For information about the contents of this release see: > > >>>>>>> > > >>>>>> > > >>>> > > >>> > > > https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=blob;f=CHANGES.txt;h=ce0aeab524505b63c7999e0371157ac2def6fe1c;hb=branch-0.8 > > >>>>>>> > > >>>>>>> Please vote on releasing this package as Apache Spark > > >>>> 0.8.1-incubating! > > >>>>>>> > > >>>>>>> The vote is open until Saturday, December 14th at 01:00 UTC and > > >>>>>>> passes if a majority of at least 3 +1 PPMC votes are cast. > > >>>>>>> > > >>>>>>> [ ] +1 Release this package as Apache Spark 0.8.1-incubating > > >>>>>>> [ ] -1 Do not release this package because ... > > >>>>>>> > > >>>>>>> To learn more about Apache Spark, please see > > >>>>>>> http://spark.incubator.apache.org/ > > >>>>>> > > >>>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> s > > >>>>> > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> s > > >>>> > > >>> > > >> > > >> > > >> > > >> -- > > >> s > > > > > > -- -- Evan Chan Staff Engineer e...@ooyala.com | <http://www.ooyala.com/> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
