I think we already set the number of partitions to 5 in tests <https://github.com/apache/spark/blob/6942aeeb0a0095a1ba85a817eb9e0edc410e5624/sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala#L60-L61> ?
On Tue, Aug 22, 2017 at 3:25 PM, Maciej Szymkiewicz <mszymkiew...@gmail.com> wrote: > Hi, > > From my experience it is possible to cut quite a lot by reducing > spark.sql.shuffle.partitions to some reasonable value (let's say > comparable to the number of cores). 200 is a serious overkill for most of > the test cases anyway. > > > Best, > Maciej > > > > On 21 August 2017 at 03:00, Dong Joon Hyun <dh...@hortonworks.com> wrote: > >> +1 for any efforts to recover Jenkins! >> >> >> >> Thank you for the direction. >> >> >> >> Bests, >> >> Dongjoon. >> >> >> >> *From: *Reynold Xin <r...@databricks.com> >> *Date: *Sunday, August 20, 2017 at 5:53 PM >> *To: *Dong Joon Hyun <dh...@hortonworks.com> >> *Cc: *"dev@spark.apache.org" <dev@spark.apache.org> >> *Subject: *Re: Increase Timeout or optimize Spark UT? >> >> >> >> It seems like it's time to look into how to cut down some of the test >> runtimes. Test runtimes will slowly go up given the way development >> happens. 3 hr is already a very long time for tests to run. >> >> >> >> >> >> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com> >> wrote: >> >> Hi, All. >> >> >> >> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has >> been hitting the build timeout. >> >> >> >> Please see the build time trend. >> >> >> >> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes >> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend >> >> >> >> All recent 22 builds fail due to timeout directly/indirectly. The last >> success (SBT with Hadoop-2.7) is 15th August. >> >> >> >> We may do the followings. >> >> >> >> 1. Increase Build Timeout (3 hr 30 min) >> 2. Optimize UTs (Scala/Java/Python/UT) >> >> >> >> But, Option 1 will be the immediate solution for now . Could you update >> the Jenkins setup? >> >> >> >> Bests, >> >> Dongjoon. >> >> >> > >
