We have the option of running extra tests in Jenkins already . Several authors are using this option for some of the changes just to be sure. Also all tests run nightly anyway. The problem is who is checking these results ? I think my point for being "neutral" about this proposal is that due diligence by contributors is more of an issue rather than the automation and adding tests to check tests itself at this point. If a PR is merged in and a nightly run failed, the authors of the merged PRs for that day should look or comment on the failure. We do see some contributors who diligently look at the results but not all. To catch these issues early is the key to stability if they failed to run the extra tests in Jenkins(or all tests manually).
Sandhya -----Original Message----- From: Qifan Chen [mailto:[email protected]] Sent: Monday, July 18, 2016 9:43 AM To: dev <[email protected]> Subject: Re: Proposal to add hive regression tests to check-PR tests The author just honestly describes the changes, and the tool picks the right tests. Thanks --Qifan On Mon, Jul 18, 2016 at 11:29 AM, Sean Broeder <[email protected]> wrote: > I'd prefer not to leave it up to authors to select which tests are > appropriate. Sometimes we get it right and others we are horribly wrong. > > Thanks, > Sean > > -----Original Message----- > From: Qifan Chen [mailto:[email protected]] > Sent: Monday, July 18, 2016 9:20 AM > To: dev <[email protected]> > Subject: Re: Proposal to add hive regression tests to check-PR tests > > I agree with Sandhya and wonder if we can enhance check-PR tests (hive > for example, in question) with the following twist. > > 1. Randomly select several (say 3) tests from regression/hive. The > rational is that we only need to sanity check the changes and a > full daily > build with test will follow the merge. > 2. Before the check-in, we always run the full regression test, and I > do > not see the value to run full Hive again in check-PR. > 3. In the future, we could find the most appreciate tests for check-PR > (instead of randomly select, or select the full set). The author > can point > out the nature of the change and the check-in tool does the selection. > For > example, a change in DoP for Hbase tables will select some tests from > regress/seabase, but not from regress/hive. > > Thanks > > --Qifan > > On Mon, Jul 18, 2016 at 10:46 AM, Sandhya Sundaresan < > [email protected]> wrote: > > > +0 for me. > > I am not sure of the need to add the whole test suite to check tests. > > The hive regressions do run nightly anyway so failures should be > > clear on each nightly run on a daily basis. > > My concern is that long running tests like hive/TEST018 are more to > test > > features like bulkload/unload and since we already have the option > > to run "extra tests" in Jenkins, I'm not sure bringing in entire > > test suites into check tests is the right approach or trend going > > forward and adding time and resources to what is supposed to be a > > sanity test > for > > every single PR. > > > > Sandhya > > > > -----Original Message----- > > From: Selva Govindarajan [mailto:[email protected]] > > Sent: Monday, July 18, 2016 7:22 AM > > To: [email protected] > > Subject: RE: Proposal to add hive regression tests to check-PR tests > > > > Hive regressions takes little less than an hour. As I said before, > > the time is not a factor because the regressions are run in parallel > > in different VMs. Seabase regressions which is run as part of > > check-PR takes around 1 hour and 40 mins. Hence hive regressions > > shouldn't add more time for check-PR to complete, but of course it > > would need another VM. > > > > Selva > > > > -----Original Message----- > > From: Jin, Jian (Seth) [mailto:[email protected]] > > Sent: Sunday, July 17, 2016 7:31 PM > > To: [email protected] > > Subject: RE: Proposal to add hive regression tests to check-PR tests > > > > How long will it take for Hive regression? > > > > Br, > > > > Seth > > > > -----Original Message----- > > From: Liu, Ming (Ming) [mailto:[email protected]] > > Sent: 2016年7月16日 9:16 > > To: [email protected] > > Subject: RE: Proposal to add hive regression tests to check-PR tests > > > > +1 to this > > > > -----Original Message----- > > From: Selva Govindarajan [mailto:[email protected]] > > Sent: Saturday, July 16, 2016 9:08 AM > > To: [email protected] > > Subject: Proposal to add hive regression tests to check-PR tests > > > > If you have subscribed to Trafodion Daily Build, you would have > > noticed that the daily build has been failing for some days. Most > > often, it is due the failure in hive regression tests run as part of > > the daily build. > > Lately, there has been some conscious effort made successfully to > > ensure that the hive regression tests can be run reliably. To > > maintain the Trafodion daily build in that state, I am proposing to > > include hive regressions to check-PR tests. It shouldn’t add the > > overall time taken to regressions tests because tests are run in > > parallel on different VMs, though it would consume more resources. > > > > > > > > - Selva > > > > > > -- > Regards, --Qifan > -- Regards, --Qifan
