For most development, you might not need to do assembly. You can run most of the unit tests if you just do sbt compile -- only the ones that spawn processes, like DistributedSuite, won't work. That said, we are looking to optimize assembly by maybe having it only package the dependencies rather than Spark itself -- there were some messages on this earlier. For now I'd just recommend doing it in a RAMFS if possible (symlink the assembly/target directory to be a RAMFS).
Matei On Oct 9, 2013, at 12:45 AM, Evan Chan <[email protected]> wrote: > Once you have compiled everything the first time using SBT (assembly will > do that for you), successive runs of assembly are much faster. I just did > it on my MacBook Pro in about 36 seconds. > > Running builds using IntelliJ or an IDE is wasted time, because the > compiled classes go to a different place than SBT. Maybe there's some way > to symlink them. > > -Evan > > > > On Tue, Oct 8, 2013 at 6:29 AM, Markus Losoi <[email protected]> wrote: > >>> Hi Markus, >> >>> have a look at the bottom of this wiki page: >> >>> https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark >> >>> IntelliJ IDEA seems to be quite popular (that I am using myself) >>> although Eclipse should work fine, too. There is another sbt plugin for >>> generating Eclipse project files. >> >> The IDE seems to work nicely, but what is the fastest way to build Spark? >> If >> I make a change to the "core" module and choose "Make Module 'core'" from >> the "Build" menu in IntelliJ Idea, then the IDE compiles the source code. >> To >> create the "spark-assembly-0.8.0-incubating-hadoop1.0.4.jar" JAR file, I >> have run "sbt assembly" on the command line. However, this takes an >> impractically long time (843 s when I last ran it on my workstation with an >> Intel Core 2 Quad Q9400 and 8 GB of RAM). Is there any faster way? >> >> Best regards, >> Markus Losoi ([email protected]) >> >> > > > -- > -- > Evan Chan > Staff Engineer > [email protected] | > > <http://www.ooyala.com/> > <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
