Matei, I wonder if we can further optimize / reduce the size of the assembly. One idea is to produce just a core assembly, and have the other projects produce their own assemblies which exclude the core dependencies.
Also, DistributedSuite is pretty slow. would it make sense to tag certain tests as the "core" tests and give it a separate build target? The overall tests that include DistributedSuite can trigger assembly, but then it would be much faster to run the core tests. -Evan On Wed, Oct 9, 2013 at 12:54 AM, Matei Zaharia <[email protected]>wrote: > For most development, you might not need to do assembly. You can run most > of the unit tests if you just do sbt compile -- only the ones that spawn > processes, like DistributedSuite, won't work. That said, we are looking to > optimize assembly by maybe having it only package the dependencies rather > than Spark itself -- there were some messages on this earlier. For now I'd > just recommend doing it in a RAMFS if possible (symlink the assembly/target > directory to be a RAMFS). > > Matei > > On Oct 9, 2013, at 12:45 AM, Evan Chan <[email protected]> wrote: > > > Once you have compiled everything the first time using SBT (assembly will > > do that for you), successive runs of assembly are much faster. I just > did > > it on my MacBook Pro in about 36 seconds. > > > > Running builds using IntelliJ or an IDE is wasted time, because the > > compiled classes go to a different place than SBT. Maybe there's some > way > > to symlink them. > > > > -Evan > > > > > > > > On Tue, Oct 8, 2013 at 6:29 AM, Markus Losoi <[email protected]> > wrote: > > > >>> Hi Markus, > >> > >>> have a look at the bottom of this wiki page: > >> > >>> > https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark > >> > >>> IntelliJ IDEA seems to be quite popular (that I am using myself) > >>> although Eclipse should work fine, too. There is another sbt plugin for > >>> generating Eclipse project files. > >> > >> The IDE seems to work nicely, but what is the fastest way to build > Spark? > >> If > >> I make a change to the "core" module and choose "Make Module 'core'" > from > >> the "Build" menu in IntelliJ Idea, then the IDE compiles the source > code. > >> To > >> create the "spark-assembly-0.8.0-incubating-hadoop1.0.4.jar" JAR file, I > >> have run "sbt assembly" on the command line. However, this takes an > >> impractically long time (843 s when I last ran it on my workstation > with an > >> Intel Core 2 Quad Q9400 and 8 GB of RAM). Is there any faster way? > >> > >> Best regards, > >> Markus Losoi ([email protected]) > >> > >> > > > > > > -- > > -- > > Evan Chan > > Staff Engineer > > [email protected] | > > > > <http://www.ooyala.com/> > > <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala > ><http://www.twitter.com/ooyala> > > -- -- Evan Chan Staff Engineer [email protected] | <http://www.ooyala.com/> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
