We need to move them to core or split the hive tests into 2 suites. SO yes code changes to test harness but too big a deal. Sandhya
-----Original Message----- From: Steve Varnau [mailto:[email protected]] Sent: Wednesday, July 27, 2016 9:24 AM To: [email protected] Subject: RE: Proposal to add hive regression tests to check-PR tests Jenkins currently runs a suite by passing the suite name to the core/sql/regress/tools/runallsb script. How would a sub-set be implemented? --Steve > -----Original Message----- > From: Qifan Chen [mailto:[email protected]] > Sent: Wednesday, July 27, 2016 9:21 AM > To: dev <[email protected]> > Subject: Re: Proposal to add hive regression tests to check-PR tests > > Sure, TEST030 can be included which can finish in 1 minute. > > TEST009 tests external tables, and TEST030 has similar tests. Maybe > exclude TEST009? > > TEST001 should be included since it tests both hive/text and hive/ORC. > > --Qifan > > > > On Wed, Jul 27, 2016 at 11:09 AM, Sandhya Sundaresan < > [email protected]> wrote: > > > Agree TEST018 is definitely a candidate. Even installer changes have > > an effect on that test. > > Will let other experts chime in for more suggestions . > > Sandhya > > > > -----Original Message----- > > From: Roberta Marton [mailto:[email protected]] > > Sent: Wednesday, July 27, 2016 8:52 AM > > To: [email protected] > > Subject: RE: Proposal to add hive regression tests to check-PR tests > > > > Perhaps we should choose tests that fail most frequently instead? > > Tests 009, 018, and 030 comes to mind. > > > > Roberta > > > > -----Original Message----- > > From: Qifan Chen [mailto:[email protected]] > > Sent: Wednesday, July 27, 2016 6:36 AM > > To: dev <[email protected]> > > Subject: Re: Proposal to add hive regression tests to check-PR tests > > > > Here is a tally of run-time (in minutes) of one run of the hive > > regression. > > > > TEST1 1 > > TEST2 1 > > TEST3 3 > > TEST5 7 > > TEST6 1 > > TEST9 4 > > TEST15 5 > > TEST17 5 > > TEST18 6 > > TEST21 1 > > TEST30 1 > > TEST31 1 > > TEST33 4 > > TEST34 2 > > TEST35 3 > > TEST36 1 > > > > The total time used is about 46 minutes. > > > > My vote will be to include a subset of from the above list what are > > fast to run (say all 1 minute ones). It probably will be a good > > idea to keep them still in hive tests. > > > > Thanks > > > > --Qifan > > > > > > On Tue, Jul 26, 2016 at 5:37 PM, Selva Govindarajan < > > [email protected]> wrote: > > > > > Thanks Steve for resurrecting this discussion. > > > > > > Hive tests have been stabilized to a greater extent that we > > > shouldn't have false failures now. Recently, there has been a > > > quite amount of contribution coming in the area related to hive in > > > Trafodion. Hence I would vote +1 for adding hive tests to check PR. > > > > > > Selva > > > > > > -----Original Message----- > > > From: Steve Varnau [mailto:[email protected]] > > > Sent: Tuesday, July 26, 2016 1:12 PM > > > To: [email protected] > > > Subject: RE: Proposal to add hive regression tests to check-PR > > > tests > > > > > > Hi, > > > > > > I wanted to revisit this discussion to come to resolution. There > > > was a digression into the idea of dynamically choosing tests, but > > > I'd like to come back to original proposal of adding an extra > > > suite to the check tests. > > > > > > As I read the thread, there were several responses in support of > > > the proposal, and a couple of reservations. The reservations > > > include increasing the chance for false failures, which already > > > can be a headache. Also the concern of adding long running tests > > > that are included in hive versus maybe adding a few more small tests > > > to core. > > > Or perhaps using "extra tests" as needed, which is available on > > > request. > > > > > > I'm willing to add another test job if that is what the community > > > wants, but might it make more sense to more small tests to core or > > > move some from hive to core? > > > > > > --Steve > > > > > > > -----Original Message----- > > > > From: Selva Govindarajan [mailto:[email protected]] > > > > Sent: Monday, July 18, 2016 10:28 PM > > > > To: [email protected] > > > > Subject: RE: Proposal to add hive regression tests to check-PR > > > > tests > > > > > > > > I totally agree with Steve to use a simple and predictable > > > > mechanism to do check PR tests, If my memory serves me right, > > > > prior to Trafodion becoming an Apache incubating project, hive > > > > tests were part of check-PR. Because of unpredictable state of > > > > hive regressions then it was decided to suspend > > > > running hive regressions as part of check-in. Based on the current > > > state > > > > of Trafodion, and the fact that the hive regressions have been > > > > stabilized to a greater extent, it is important that this > > > > stability is maintained by the future contributions. Recently > > > > many contributions have come in hive-related area of the code. > > > > > > > > Adding hive regressions as part of check-PR should not increase > > > > the overall time to complete the check-PR, but it would require > > > > additional resources,. > > > > Hence, Trafodion Jenkins infrastructure would incur additional cost. > > > > > > > > I am expecting the Trafodion Release Manager of R2.1 will help > > > > us to determine with the community input what is the best option > > > > to go with. > > > > > > > > Selva > > > > -----Original Message----- > > > > From: Steve Varnau [mailto:[email protected]] > > > > Sent: Monday, July 18, 2016 12:27 PM > > > > To: [email protected] > > > > Subject: RE: Proposal to add hive regression tests to check-PR > > > > tests > > > > > > > > The current test process looks at which files have been modified > > > > and puts it into a bucket, which is used to determine what tests to > > > > run. > > > > However, the only buckets that now exist are DOC and NONDOC. > > > > > > > > So if the change consists only of things in the docs/ tree, then > > > > it only does static check and a docs build. If there are > > > > non-docs changes, it assumes it needs to run all the build and > > > > tests. > > > > > > > > It is pretty conservative, but the more heuristics we put in to > > > > customize the tests, the more chance that it will miss > > > > something. I can imagine a connectivity only change not running > > > > the jobs that don't exercise connectivity. But figuring out what > > > > things might affect hive tests seems much harder. > > > > > > > > There are many things (installer, executor,...) that might > > > > affect any of our tests. Seems safer to keep the test > > > > heuristics very simple and predictable, and change the content > > > > of the test suites to what ought to be in check versus nightly. > > > > > > > > --Steve > > > > > > > > > -----Original Message----- > > > > > From: Qifan Chen [mailto:[email protected]] > > > > > Sent: Monday, July 18, 2016 9:43 AM > > > > > To: dev <[email protected]> > > > > > Subject: Re: Proposal to add hive regression tests to check-PR > > > > > tests > > > > > > > > > > The author just honestly describes the changes, and the tool > > > > > picks the right tests. Thanks --Qifan > > > > > > > > > > On Mon, Jul 18, 2016 at 11:29 AM, Sean Broeder > > > > > <[email protected]> > > > > > wrote: > > > > > > > > > > > I'd prefer not to leave it up to authors to select which > > > > > > tests are appropriate. Sometimes we get it right and others > > > > > > we are horribly wrong. > > > > > > > > > > > > Thanks, > > > > > > Sean > > > > > > > > > > > > -----Original Message----- > > > > > > From: Qifan Chen [mailto:[email protected]] > > > > > > Sent: Monday, July 18, 2016 9:20 AM > > > > > > To: dev <[email protected]> > > > > > > Subject: Re: Proposal to add hive regression tests to > > > > > > check-PR tests > > > > > > > > > > > > I agree with Sandhya and wonder if we can enhance check-PR > > > > > > tests (hive > > > > > for > > > > > > example, in question) with the following twist. > > > > > > > > > > > > 1. Randomly select several (say 3) tests from > > > > > > regression/hive. > > > > > > The > > > > > > rational is that we only need to sanity check the > > > > > > changes and a full daily > > > > > > build with test will follow the merge. > > > > > > 2. Before the check-in, we always run the full regression > > > > > > test, and I do > > > > > > not see the value to run full Hive again in check-PR. > > > > > > 3. In the future, we could find the most appreciate tests > > > > > > for check-PR > > > > > > (instead of randomly select, or select the full set). > > > > > > The author can point > > > > > > out the nature of the change and the check-in tool does > > > > > > the selection. > > > > > > For > > > > > > example, a change in DoP for Hbase tables will select > > > > > > some tests from > > > > > > regress/seabase, but not from regress/hive. > > > > > > > > > > > > Thanks > > > > > > > > > > > > --Qifan > > > > > > > > > > > > On Mon, Jul 18, 2016 at 10:46 AM, Sandhya Sundaresan < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > +0 for me. > > > > > > > I am not sure of the need to add the whole test suite to > > > > > > > check tests. > > > > > > > The hive regressions do run nightly anyway so failures > > > > > > > should be clear on each nightly run on a daily basis. > > > > > > > My concern is that long running tests like hive/TEST018 are > > more > > > > > > > to > > > > > > test > > > > > > > features like bulkload/unload and since we already have > > > > > > > the option to run "extra tests" in Jenkins, I'm not sure > > > > > > > bringing in entire test suites into check tests is the > > > > > > > right approach or trend going forward and adding time and > > > > > > > resources to what is supposed to be a sanity test > > > > > > for > > > > > > > every single PR. > > > > > > > > > > > > > > Sandhya > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Selva Govindarajan > > > > > > > [mailto:[email protected]] > > > > > > > Sent: Monday, July 18, 2016 7:22 AM > > > > > > > To: [email protected] > > > > > > > Subject: RE: Proposal to add hive regression tests to > > > > > > > check-PR tests > > > > > > > > > > > > > > Hive regressions takes little less than an hour. As I said > > > > > > > before, the time is not a factor because the regressions > > > > > > > are run in parallel in different VMs. Seabase regressions > > > > > > > which is run as part of check-PR takes around 1 hour and > > > > > > > 40 mins. Hence hive regressions shouldn't add more time > > > > > > > for check-PR to complete, but of course it would need another > > > > > > > VM. > > > > > > > > > > > > > > Selva > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Jin, Jian (Seth) [mailto:[email protected]] > > > > > > > Sent: Sunday, July 17, 2016 7:31 PM > > > > > > > To: [email protected] > > > > > > > Subject: RE: Proposal to add hive regression tests to > > > > > > > check-PR tests > > > > > > > > > > > > > > How long will it take for Hive regression? > > > > > > > > > > > > > > Br, > > > > > > > > > > > > > > Seth > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Liu, Ming (Ming) [mailto:[email protected]] > > > > > > > Sent: 2016年7月16日 9:16 > > > > > > > To: [email protected] > > > > > > > Subject: RE: Proposal to add hive regression tests to > > > > > > > check-PR tests > > > > > > > > > > > > > > +1 to this > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Selva Govindarajan > > > > > > > [mailto:[email protected]] > > > > > > > Sent: Saturday, July 16, 2016 9:08 AM > > > > > > > To: [email protected] > > > > > > > Subject: Proposal to add hive regression tests to check-PR > > > > > > > tests > > > > > > > > > > > > > > If you have subscribed to Trafodion Daily Build, you would > > > > > > > have noticed that the daily build has been failing for > > > > > > > some days. > > > > > > > Most often, it is due the failure in hive regression tests > > > > > > > run as part of the daily build. > > > > > > > Lately, there has been some conscious effort made > > > > > > > successfully to ensure that the hive regression tests can be > > > > > > > run reliably. > > > > > > > To maintain the Trafodion daily build in that state, I am > > > > > > > proposing to include hive regressions to check-PR tests. > > > > > > > It shouldn’t add the overall time taken to regressions > > > > > > > tests because tests are run in parallel on different VMs, > > > > > > > though it would consume more resources. > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Selva > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, --Qifan > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, --Qifan > > > > > > > > > > > -- > > Regards, --Qifan > > > > > > -- > Regards, --Qifan
