I fully support this. A smoothly running test infrastructure helps
everybody’s work just flow better.

The Jenkins Pull Request Builder is mostly functioning
again. However, we are working on a simpler technical pipeline for
testing patches, as this plug-in has been a constant source of
downtime and issues for us, and is very hard to debug.

Yep. One such issue that happens too often is that Jenkins simply fails to
fetch from git. Hopefully a new pipeline will be able to fetch more
reliably.

flaky tests

Dunno if these were some of the ones recently fixed, but the flakiest tests
seem to be the Kafka and Flume tests in Spark Streaming, based purely on my
subjective experience. It would be great if we could stabilize them!

Time of tests

PSA: Here are some related JIRA issues for those interested in working on
our testing setup:

   - SPARK-3431: Parallelize execution of tests
   <https://issues.apache.org/jira/browse/SPARK-3431>
   - SPARK-3432: Fix logging of unit test execution time
   <https://issues.apache.org/jira/browse/SPARK-3432>

Nick
​

On Sun, Sep 14, 2014 at 2:20 AM, Josh Rosen <rosenvi...@gmail.com> wrote:

> Also, huge thanks to Cheng Lian, who tracked down and fixed the final
> issue that was causing the Maven master build’s Spark SQL tests to fail!
>
> On September 13, 2014 at 11:08:00 PM, Patrick Wendell (pwend...@gmail.com)
> wrote:
> Hey All,
>
> Wanted to send a quick update about test infrastructure. With the
> number of contributors we have and the rate of development,
> maintaining a well-oiled test infra is really important.
>
> Every time a flaky test fails a legitimate pull request, it wastes
> developer time and effort.
>
> 1. Master build: Spark's master builds are back to green again in
> Maven and SBT after a long time of instability. Big thanks to Josh
> Rosen, Andrew Or, Nick Chammas, Shane Knapp, Sean Owen, and many
> others who were involved in pinpointing and fixing fairly convoluted
> test failure issues.
>
> 2. Jenkins PRB: The Jenkins Pull Request Builder is mostly functioning
> again. However, we are working on a simpler technical pipeline for
> testing patches, as this plug-in has been a constant source of
> downtime and issues for us, and is very hard to debug.
>
> 3. Reverting flaky patches: Going forward - we may revert patches that
> seem to be the root cause of flaky or failing tests. This is necessary
> as these days, the test infra being down will block something like
> 10-30 in-flight patches on a given day. This puts the onus back on the
> test writer to try and figure out what's going on - we'll of course
> help debug the issue!
>
> 4. Time of tests: With hundreds (thousands?) of tests, we will have a
> very high bar for tests which take several seconds or longer. Things
> like Thread.sleep() bloat test time when proper synchronization
> mechanisms should be used. Expect reviewers to push back on any
> long-running tests, in many cases they can be re-written to be both
> shorter and better.
>
> Thanks again to everyone putting in effort on this, we've made a ton
> of progress in the last few weeks. A solid test infra will help us
> scale and move quickly as Spark development continues to accelerate.
>
> - Patrick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Reply via email to