Re: Spark build time
I suggest searching the archives for this list as there were several previous discussions about this problem. JIRA also has several issues related to this. Some pointers: - SPARK-3431 https://issues.apache.org/jira/browse/SPARK-3431: Parallelize Scala/Java test execution - http://apache-spark-developers-list.1001551.n3.nabble.com/Unit-tests-in-lt-5-minutes-td7757.html - SPARK-4746 https://issues.apache.org/jira/browse/SPARK-4746: integration tests should be separated from faster unit tests - http://apache-spark-developers-list.1001551.n3.nabble.com/Building-Spark-with-Pants-td10397.html Summary is, everyone agrees the long times are a problem and wants the build and tests to run faster. There are several things that can be done, but they all require a lot of work. Nick On Wed, Apr 22, 2015 at 5:25 AM Olivier Girardot ssab...@gmail.com wrote: I agree, it's what I did :) I was just wondering if it was considered a problem or something to work on, I personally think so because the feedback loop should be as quick as possible, and therefore if there was someone I could help. Le mar. 21 avr. 2015 à 22:20, Reynold Xin r...@databricks.com a écrit : It runs tons of integration tests. I think most developers just let Jenkins run the full suite of them. On Tue, Apr 21, 2015 at 12:54 PM, Olivier Girardot ssab...@gmail.com wrote: Hi everyone, I was just wandering about the Spark full build time (including tests), 1h48 seems to me quite... spacious. What's taking most of the time ? Is the build mainly integration tests ? Is there any roadmap or jiras dedicated to that we can chip in ? Regards, Olivier.
Re: Spark build time
I agree, it's what I did :) I was just wondering if it was considered a problem or something to work on, I personally think so because the feedback loop should be as quick as possible, and therefore if there was someone I could help. Le mar. 21 avr. 2015 à 22:20, Reynold Xin r...@databricks.com a écrit : It runs tons of integration tests. I think most developers just let Jenkins run the full suite of them. On Tue, Apr 21, 2015 at 12:54 PM, Olivier Girardot ssab...@gmail.com wrote: Hi everyone, I was just wandering about the Spark full build time (including tests), 1h48 seems to me quite... spacious. What's taking most of the time ? Is the build mainly integration tests ? Is there any roadmap or jiras dedicated to that we can chip in ? Regards, Olivier.
RE: Spark build time
If you are using MVN there are some parameters (MAVEN_OPTS) which need to be set in order to give the underlying environment enough memory. See the instructions here: https://spark.apache.org/docs/latest/building-spark.html -Original Message- From: Reynold Xin r...@databricks.com Sent: 4/21/2015 4:21 PM To: Olivier Girardot ssab...@gmail.com Cc: dev@spark.apache.org dev@spark.apache.org Subject: Re: Spark build time It runs tons of integration tests. I think most developers just let Jenkins run the full suite of them. On Tue, Apr 21, 2015 at 12:54 PM, Olivier Girardot ssab...@gmail.com wrote: Hi everyone, I was just wandering about the Spark full build time (including tests), 1h48 seems to me quite... spacious. What's taking most of the time ? Is the build mainly integration tests ? Is there any roadmap or jiras dedicated to that we can chip in ? Regards, Olivier.
Re: Spark build time
It runs tons of integration tests. I think most developers just let Jenkins run the full suite of them. On Tue, Apr 21, 2015 at 12:54 PM, Olivier Girardot ssab...@gmail.com wrote: Hi everyone, I was just wandering about the Spark full build time (including tests), 1h48 seems to me quite... spacious. What's taking most of the time ? Is the build mainly integration tests ? Is there any roadmap or jiras dedicated to that we can chip in ? Regards, Olivier.
