Re: python test infrastructure

2018-09-06 Thread Imran Rashid
> On Wed, Sep 5, 2018 at 11:59 PM Hyukjin Kwon wrote: > > > > > 1. all of the output in target/test-reports & python/unit-tests.log should be included in the jenkins archived artifacts. > > > > Hmmm, I thought they are already archived ( https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequ

Re: python test infrastructure

2018-09-05 Thread Hyukjin Kwon
> 1. all of the output in target/test-reports & python/unit-tests.log should be included in the jenkins archived artifacts. Hmmm, I thought they are already archived ( https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95734/artifact/target/unit-tests.log ). FWIW, unit-tests.log ar

Re: python test infrastructure

2018-09-05 Thread Imran Rashid
one more: seems like python/run-tests should have an option at least to not bail at the first failure: https://github.com/apache/spark/blob/master/python/run-tests.py#L113-L132 this is particularly annoying with flaky tests -- since the rest of the tests aren't run, you don't know whether you *onl

python test infrastructure

2018-09-05 Thread Imran Rashid
Hi all, More pyspark noob questions from me. I find it really hard to figure out what versions of python I should be testing and what is tested upstream. While I'd like to just know the answers to those questions, more importantly I'd like to make sure that info is visible somewhere so all devs c