I suggest searching the archives for this list as there were several previous discussions about this problem. JIRA also has several issues related to this.
Some pointers: - SPARK-3431 <https://issues.apache.org/jira/browse/SPARK-3431>: Parallelize Scala/Java test execution - http://apache-spark-developers-list.1001551.n3.nabble.com/Unit-tests-in-lt-5-minutes-td7757.html - SPARK-4746 <https://issues.apache.org/jira/browse/SPARK-4746>: integration tests should be separated from faster unit tests - http://apache-spark-developers-list.1001551.n3.nabble.com/Building-Spark-with-Pants-td10397.html Summary is, everyone agrees the long times are a problem and wants the build and tests to run faster. There are several things that can be done, but they all require a lot of work. Nick On Wed, Apr 22, 2015 at 5:25 AM Olivier Girardot <ssab...@gmail.com> wrote: > I agree, it's what I did :) > I was just wondering if it was considered a "problem" or something to work > on, I personally think so because the feedback loop should be as quick as > possible, and therefore if there was someone I could help. > > Le mar. 21 avr. 2015 à 22:20, Reynold Xin <r...@databricks.com> a écrit : > > > It runs tons of integration tests. I think most developers just let > > Jenkins run the full suite of them. > > > > On Tue, Apr 21, 2015 at 12:54 PM, Olivier Girardot <ssab...@gmail.com> > > wrote: > > > >> Hi everyone, > >> I was just wandering about the Spark full build time (including tests), > >> 1h48 seems to me quite... spacious. What's taking most of the time ? Is > >> the > >> build mainly integration tests ? Is there any roadmap or jiras dedicated > >> to > >> that we can chip in ? > >> > >> Regards, > >> > >> Olivier. > >> > > > > >
