If you retire the wiki page MiniMR and PTest2
<https://cwiki.apache.org/confluence/display/Hive/MiniMR+and+PTest2> then
five links from other docs will have to be removed:
Page: HiveDeveloperFAQ
<https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ>
Page: TestingDocs
<https://cwiki.apache.org/confluence/display/Hive/TestingDocs>
Home page: Home <https://cwiki.apache.org/confluence/display/Hive/Home>
Page: Hive PreCommit Patch Testing
<https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing>
Page: DeveloperDocs
<https://cwiki.apache.org/confluence/display/Hive/DeveloperDocs>
-- Lefty
On Mon, Jul 14, 2014 at 12:58 AM, Szehon Ho <sze...@cloudera.com> wrote:
> Hi,
>
> This is now done, with some help from Gunther the Pre-commit test framework
> pick from the itests/qtest/testconfiguration.properties to find the
> MiniXCliDriver tests, same as the normal test runner. New tests are picked
> automatically, no need to do as mentioned above (and we can probably retire
> that wiki page).
>
> There are just 1-2 failing MiniXCliDriver tests that hasn't been run as
> part of pre-commit suite until this, that may show up in the failures now.
>
> Thanks
> Szehon
>
>
>
>
>
>
> On Thu, Jun 19, 2014 at 7:09 AM, Szehon Ho <sze...@cloudera.com> wrote:
>
> > (changing subject)
> >
> > The MiniTezCliDriver tests have timed-out lately in the pre-commit tests,
> > reducing coverage of the test as Ashutosh reported. I now configured the
> > parallel-test framework to run MiniTezCliDriver in batches of 15 qtest,
> > like the others. Now the timeout issue is fixed, and test reports are
> > showing up for those.
> >
> > A nice thing is it speeds up the average speed of pre-commit tests by a
> > lot, as it was bottlenecked on running all the 79 MiniTezCliDriver tests
> on
> > one node.
> >
> > The only impact is, now if you are adding new MiniTezCliDriver tests,
> they
> > need to be manually added in the Ptest config on the build machine , like
> > explained in:
> > https://cwiki.apache.org/confluence/display/Hive/MiniMR+and+PTest2.
> I've
> > added all 79 current tests manually. It might be a bigger impact for
> this
> > driver than others, as Hive-Tez is under heavy development. I filed
> > HIVE-7254 <https://issues.apache.org/jira/browse/HIVE-7254> to explore
> > improving it, but for now please follow that or notify me, to add the new
> > test to the pre-commit test coverage.
> >
> > Thanks
> > Szehon
> >
> >
> >
> > On Fri, Jun 13, 2014 at 3:16 PM, Brock Noland <br...@cloudera.com>
> wrote:
> >
> >> + dev
> >>
> >> Good call, yep that will need to be configured.
> >>
> >> Brock
> >>
> >>
> >> On Fri, Jun 13, 2014 at 10:29 AM, Szehon Ho <sze...@cloudera.com>
> wrote:
> >>
> >>> I was studying this a bit more, I believe the MiniTezCliDriver tests
> are
> >>> hitting timeout after 2 hours as error code is 124. The framework is
> >>> running all of them in one call, I'll try to chunk the tests into
> batches
> >>> like the other q-tests.
> >>>
> >>> I'll try to take a look next week at this.
> >>>
> >>> Thanks
> >>> Szehon
> >>>
> >>>
> >>> On Mon, Jun 9, 2014 at 1:13 PM, Szehon Ho <sze...@cloudera.com> wrote:
> >>>
> >>>> It looks like JVM OOM crash during MiniTezCliDriver tests, or its
> >>>> otherwise crashing. The 407 log has failures, but the 408 log is cut
> off.
> >>>>
> >>>>
> >>>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-407/failed/TestMiniTezCliDriver/maven-test.txt
> >>>>
> >>>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-408/failed/TestMiniTezCliDriver/maven-test.txt
> >>>>
> >>>> The MAVEN_OPTS is already set to "-XmX2g -XX:MaxPermSize=256M". Do
> you
> >>>> guys know of any such issues?
> >>>>
> >>>> Thanks,
> >>>> Szehon
> >>>>
> >>>>
> >>>>
> >>>> On Sun, Jun 8, 2014 at 12:05 PM, Brock Noland <br...@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> Looks like it's failing to generate a to generate a test output:
> >>>>>
> >>>>>
> >>>>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-408/failed/TestMiniTezCliDriver/
> >>>>>
> >>>>>
> >>>>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-408/failed/TestMiniTezCliDriver/TestMiniTezCliDriver.txt
> >>>>>
> >>>>> exiting with 124 here:
> >>>>>
> >>>>> + wait 21961
> >>>>> + timeout 2h mvn -B -o test
> -Dmaven.repo.local=/home/hiveptest//ip-10-31-188-232-hiveptest-2/maven
> -Phadoop-2 -Phadoop-2 -Dtest=TestMiniTezCliDriver
> >>>>> + ret=124
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun, Jun 8, 2014 at 11:25 AM, Ashutosh Chauhan <
> >>>>> hashut...@apache.org> wrote:
> >>>>>
> >>>>>> Build #407 ran MiniTezCliDriver
> >>>>>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/407/testReport/org.apache.hadoop.hive.cli/
> >>>>>>
> >>>>>> but Build #408 didn't
> >>>>>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/408/testReport/org.apache.hadoop.hive.cli/
> >>>>>>
> >>>>>>
> >>>>>> On Sat, Jun 7, 2014 at 12:25 PM, Szehon Ho <sze...@cloudera.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Sounds like there's randomness, either in PTest test-parser or in
> >>>>>>> the maven test itself. In the history now, its running between
> 5633-5707,
> >>>>>>> which is similar to your range.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/394/testReport/history/
> >>>>>>>
> >>>>>>> I didnt see any in history without MiniTezCLIDriver, can you point
> >>>>>>> me to a build no. if you see one? If nobody else knows
> immediately, I can
> >>>>>>> dig deeper at it next week to try to find out.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sat, Jun 7, 2014 at 9:00 AM, Ashutosh Chauhan <
> >>>>>>> hashut...@apache.org> wrote:
> >>>>>>>
> >>>>>>>> I noticed that PTest2 framework runs different number of tests on
> >>>>>>>> various runs. e.g., on yesterday's runs I saw it ran 5585 & 5510
> tests on
> >>>>>>>> subsequent runs. In particular, it seems its running
> MiniTezCliDriver tests
> >>>>>>>> in only half the runs. Anyone observed this?
> >>>>>>>>
> >>>>>>>>
> >
>
