Agree TEST018 is definitely a candidate. Even installer changes have an effect on that test. Will let other experts chime in for more suggestions . Sandhya
-----Original Message----- From: Roberta Marton [mailto:[email protected]] Sent: Wednesday, July 27, 2016 8:52 AM To: [email protected] Subject: RE: Proposal to add hive regression tests to check-PR tests Perhaps we should choose tests that fail most frequently instead? Tests 009, 018, and 030 comes to mind. Roberta -----Original Message----- From: Qifan Chen [mailto:[email protected]] Sent: Wednesday, July 27, 2016 6:36 AM To: dev <[email protected]> Subject: Re: Proposal to add hive regression tests to check-PR tests Here is a tally of run-time (in minutes) of one run of the hive regression. TEST1 1 TEST2 1 TEST3 3 TEST5 7 TEST6 1 TEST9 4 TEST15 5 TEST17 5 TEST18 6 TEST21 1 TEST30 1 TEST31 1 TEST33 4 TEST34 2 TEST35 3 TEST36 1 The total time used is about 46 minutes. My vote will be to include a subset of from the above list what are fast to run (say all 1 minute ones). It probably will be a good idea to keep them still in hive tests. Thanks --Qifan On Tue, Jul 26, 2016 at 5:37 PM, Selva Govindarajan < [email protected]> wrote: > Thanks Steve for resurrecting this discussion. > > Hive tests have been stabilized to a greater extent that we shouldn't > have false failures now. Recently, there has been a quite amount of > contribution coming in the area related to hive in Trafodion. Hence I > would vote +1 for adding hive tests to check PR. > > Selva > > -----Original Message----- > From: Steve Varnau [mailto:[email protected]] > Sent: Tuesday, July 26, 2016 1:12 PM > To: [email protected] > Subject: RE: Proposal to add hive regression tests to check-PR tests > > Hi, > > I wanted to revisit this discussion to come to resolution. There was > a digression into the idea of dynamically choosing tests, but I'd like > to come back to original proposal of adding an extra suite to the > check tests. > > As I read the thread, there were several responses in support of the > proposal, and a couple of reservations. The reservations include > increasing the chance for false failures, which already can be a > headache. Also the concern of adding long running tests that are > included in hive versus maybe adding a few more small tests to core. > Or perhaps using "extra tests" as needed, which is available on request. > > I'm willing to add another test job if that is what the community > wants, but might it make more sense to more small tests to core or > move some from hive to core? > > --Steve > > > -----Original Message----- > > From: Selva Govindarajan [mailto:[email protected]] > > Sent: Monday, July 18, 2016 10:28 PM > > To: [email protected] > > Subject: RE: Proposal to add hive regression tests to check-PR tests > > > > I totally agree with Steve to use a simple and predictable mechanism > > to do check PR tests, If my memory serves me right, prior to > > Trafodion becoming an Apache incubating project, hive tests were > > part of check-PR. Because of unpredictable state of hive regressions > > then it was decided to suspend > > running hive regressions as part of check-in. Based on the current > state > > of Trafodion, and the fact that the hive regressions have been > > stabilized to a greater extent, it is important that this stability > > is maintained by the future contributions. Recently many > > contributions have come in hive-related area of the code. > > > > Adding hive regressions as part of check-PR should not increase the > > overall time to complete the check-PR, but it would require > > additional resources,. > > Hence, Trafodion Jenkins infrastructure would incur additional cost. > > > > I am expecting the Trafodion Release Manager of R2.1 will help us to > > determine with the community input what is the best option to go with. > > > > Selva > > -----Original Message----- > > From: Steve Varnau [mailto:[email protected]] > > Sent: Monday, July 18, 2016 12:27 PM > > To: [email protected] > > Subject: RE: Proposal to add hive regression tests to check-PR tests > > > > The current test process looks at which files have been modified and > > puts it into a bucket, which is used to determine what tests to run. > > However, the only buckets that now exist are DOC and NONDOC. > > > > So if the change consists only of things in the docs/ tree, then it > > only does static check and a docs build. If there are non-docs > > changes, it assumes it needs to run all the build and tests. > > > > It is pretty conservative, but the more heuristics we put in to > > customize the tests, the more chance that it will miss something. I > > can imagine a connectivity only change not running the jobs that > > don't exercise connectivity. But figuring out what things might > > affect hive tests seems much harder. > > > > There are many things (installer, executor,...) that might affect > > any of our tests. Seems safer to keep the test heuristics very > > simple and predictable, and change the content of the test suites to > > what ought to be in check versus nightly. > > > > --Steve > > > > > -----Original Message----- > > > From: Qifan Chen [mailto:[email protected]] > > > Sent: Monday, July 18, 2016 9:43 AM > > > To: dev <[email protected]> > > > Subject: Re: Proposal to add hive regression tests to check-PR > > > tests > > > > > > The author just honestly describes the changes, and the tool picks > > > the right tests. Thanks --Qifan > > > > > > On Mon, Jul 18, 2016 at 11:29 AM, Sean Broeder > > > <[email protected]> > > > wrote: > > > > > > > I'd prefer not to leave it up to authors to select which tests > > > > are appropriate. Sometimes we get it right and others we are > > > > horribly wrong. > > > > > > > > Thanks, > > > > Sean > > > > > > > > -----Original Message----- > > > > From: Qifan Chen [mailto:[email protected]] > > > > Sent: Monday, July 18, 2016 9:20 AM > > > > To: dev <[email protected]> > > > > Subject: Re: Proposal to add hive regression tests to check-PR > > > > tests > > > > > > > > I agree with Sandhya and wonder if we can enhance check-PR tests > > > > (hive > > > for > > > > example, in question) with the following twist. > > > > > > > > 1. Randomly select several (say 3) tests from regression/hive. > > > > The > > > > rational is that we only need to sanity check the changes > > > > and a full daily > > > > build with test will follow the merge. > > > > 2. Before the check-in, we always run the full regression > > > > test, and I do > > > > not see the value to run full Hive again in check-PR. > > > > 3. In the future, we could find the most appreciate tests for > > > > check-PR > > > > (instead of randomly select, or select the full set). The > > > > author can point > > > > out the nature of the change and the check-in tool does the > > > > selection. > > > > For > > > > example, a change in DoP for Hbase tables will select some > > > > tests from > > > > regress/seabase, but not from regress/hive. > > > > > > > > Thanks > > > > > > > > --Qifan > > > > > > > > On Mon, Jul 18, 2016 at 10:46 AM, Sandhya Sundaresan < > > > > [email protected]> wrote: > > > > > > > > > +0 for me. > > > > > I am not sure of the need to add the whole test suite to > > > > > check tests. > > > > > The hive regressions do run nightly anyway so failures should > > > > > be clear on each nightly run on a daily basis. > > > > > My concern is that long running tests like hive/TEST018 are more > > > > > to > > > > test > > > > > features like bulkload/unload and since we already have the > > > > > option to run "extra tests" in Jenkins, I'm not sure bringing > > > > > in entire test suites into check tests is the right approach > > > > > or trend going forward and adding time and resources to what > > > > > is supposed to be a sanity test > > > > for > > > > > every single PR. > > > > > > > > > > Sandhya > > > > > > > > > > -----Original Message----- > > > > > From: Selva Govindarajan [mailto:[email protected]] > > > > > Sent: Monday, July 18, 2016 7:22 AM > > > > > To: [email protected] > > > > > Subject: RE: Proposal to add hive regression tests to check-PR > > > > > tests > > > > > > > > > > Hive regressions takes little less than an hour. As I said > > > > > before, the time is not a factor because the regressions are > > > > > run in parallel in different VMs. Seabase regressions which is > > > > > run as part of check-PR takes around 1 hour and 40 mins. Hence > > > > > hive regressions shouldn't add more time for check-PR to > > > > > complete, but of course it would need another VM. > > > > > > > > > > Selva > > > > > > > > > > -----Original Message----- > > > > > From: Jin, Jian (Seth) [mailto:[email protected]] > > > > > Sent: Sunday, July 17, 2016 7:31 PM > > > > > To: [email protected] > > > > > Subject: RE: Proposal to add hive regression tests to check-PR > > > > > tests > > > > > > > > > > How long will it take for Hive regression? > > > > > > > > > > Br, > > > > > > > > > > Seth > > > > > > > > > > -----Original Message----- > > > > > From: Liu, Ming (Ming) [mailto:[email protected]] > > > > > Sent: 2016年7月16日 9:16 > > > > > To: [email protected] > > > > > Subject: RE: Proposal to add hive regression tests to check-PR > > > > > tests > > > > > > > > > > +1 to this > > > > > > > > > > -----Original Message----- > > > > > From: Selva Govindarajan [mailto:[email protected]] > > > > > Sent: Saturday, July 16, 2016 9:08 AM > > > > > To: [email protected] > > > > > Subject: Proposal to add hive regression tests to check-PR > > > > > tests > > > > > > > > > > If you have subscribed to Trafodion Daily Build, you would > > > > > have noticed that the daily build has been failing for some days. > > > > > Most often, it is due the failure in hive regression tests run > > > > > as part of the daily build. > > > > > Lately, there has been some conscious effort made successfully > > > > > to ensure that the hive regression tests can be run reliably. > > > > > To maintain the Trafodion daily build in that state, I am > > > > > proposing to include hive regressions to check-PR tests. It > > > > > shouldn’t add the overall time taken to regressions tests > > > > > because tests are run in parallel on different VMs, though it > > > > > would consume more resources. > > > > > > > > > > > > > > > > > > > > - Selva > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, --Qifan > > > > > > > > > > > > > > > > -- > > > Regards, --Qifan > -- Regards, --Qifan
