Re: Splitting tests in PTest2

[Szehon Ho]

Yes thanks for looking into it.  Can you edit testconfiguration.properties
to add a section for LlapCliDriver tests, see HIVE-7254 for details.  That
way the framework can identify which tests it needs to run and then split
it into parallel batches across test nodes.  Otherwise it will just run all
of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one huge
batch.

Once you finish, let me know and I'll tweak the test properties to set a
batch size for that one.

Thanks
Szehon

On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran  wrote:

> Hello everyone
>
> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split across
> multiple nodes. I can see from the logs that MiniTezCliDriver are broken
> down into many tests and ran on different machines.
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> <
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> >
>
> How can we do the same for MiniLlapCliDriver tests? Do we need to do
> something in PTest2 to enable splitting of tests?
> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>
> Thanks
> Prasanth

Re: Splitting tests in PTest2

[Prasanth Jayachandran]

Thanks Szehon! I will look at it and update you once done.

> On Sep 28, 2015, at 6:37 PM, Szehon Ho  wrote:
> 
> Yes thanks for looking into it.  Can you edit testconfiguration.properties
> to add a section for LlapCliDriver tests, see HIVE-7254 for details.  That
> way the framework can identify which tests it needs to run and then split
> it into parallel batches across test nodes.  Otherwise it will just run all
> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one huge
> batch.
> 
> Once you finish, let me know and I'll tweak the test properties to set a
> batch size for that one.
> 
> Thanks
> Szehon
> 
> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran > wrote:
> 
>> Hello everyone
>> 
>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split across
>> multiple nodes. I can see from the logs that MiniTezCliDriver are broken
>> down into many tests and ran on different machines.
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>> <
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>>> 
>> 
>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>> something in PTest2 to enable splitting of tests?
>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>> 
>> Thanks
>> Prasanth

Re: Splitting tests in PTest2

[Prasanth Jayachandran]

@Szehon.. MiniLlapCliDriver  uses the same set of files from minitez. Can that 
be reused as such without duplicating?


> On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran 
>  wrote:
> 
> Thanks Szehon! I will look at it and update you once done.
> 
>> On Sep 28, 2015, at 6:37 PM, Szehon Ho  wrote:
>> 
>> Yes thanks for looking into it.  Can you edit testconfiguration.properties
>> to add a section for LlapCliDriver tests, see HIVE-7254 for details.  That
>> way the framework can identify which tests it needs to run and then split
>> it into parallel batches across test nodes.  Otherwise it will just run all
>> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one huge
>> batch.
>> 
>> Once you finish, let me know and I'll tweak the test properties to set a
>> batch size for that one.
>> 
>> Thanks
>> Szehon
>> 
>> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran >> wrote:
>> 
>>> Hello everyone
>>> 
>>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split across
>>> multiple nodes. I can see from the logs that MiniTezCliDriver are broken
>>> down into many tests and ran on different machines.
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>>> <
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
 
>>> 
>>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>>> something in PTest2 to enable splitting of tests?
>>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>>> 
>>> Thanks
>>> Prasanth
> 

Re: Splitting tests in PTest2

[Szehon Ho]

OK it may be possible, I will take a look!

Thanks
Szehon

On Mon, Sep 28, 2015 at 4:56 PM, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> @Szehon.. MiniLlapCliDriver  uses the same set of files from minitez. Can
> that be reused as such without duplicating?
>
>
> > On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
> >
> > Thanks Szehon! I will look at it and update you once done.
> >
> >> On Sep 28, 2015, at 6:37 PM, Szehon Ho  wrote:
> >>
> >> Yes thanks for looking into it.  Can you edit
> testconfiguration.properties
> >> to add a section for LlapCliDriver tests, see HIVE-7254 for details.
> That
> >> way the framework can identify which tests it needs to run and then
> split
> >> it into parallel batches across test nodes.  Otherwise it will just run
> all
> >> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one
> huge
> >> batch.
> >>
> >> Once you finish, let me know and I'll tweak the test properties to set a
> >> batch size for that one.
> >>
> >> Thanks
> >> Szehon
> >>
> >> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <
> prasan...@apache.org
> >>> wrote:
> >>
> >>> Hello everyone
> >>>
> >>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split
> across
> >>> multiple nodes. I can see from the logs that MiniTezCliDriver are
> broken
> >>> down into many tests and ran on different machines.
> >>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> >>> <
> >>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> 
> >>>
> >>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
> >>> something in PTest2 to enable splitting of tests?
> >>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
> >>>
> >>> Thanks
> >>> Prasanth
> >
>
>

Re: Splitting tests in PTest2

[Szehon Ho]

OK so there is a bit of a chicken-egg problem, I cannot change it until
11642 is in, otherwise it will break the other builds.

Once its in I can update the properties file to reference the new
LlapCliDriver.  Maybe you can check it in and I will try enabling it
tonight.

On Mon, Sep 28, 2015 at 5:05 PM, Szehon Ho  wrote:

> OK it may be possible, I will take a look!
>
> Thanks
> Szehon
>
> On Mon, Sep 28, 2015 at 4:56 PM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
>
>> @Szehon.. MiniLlapCliDriver  uses the same set of files from minitez. Can
>> that be reused as such without duplicating?
>>
>>
>> > On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran <
>> pjayachand...@hortonworks.com> wrote:
>> >
>> > Thanks Szehon! I will look at it and update you once done.
>> >
>> >> On Sep 28, 2015, at 6:37 PM, Szehon Ho  wrote:
>> >>
>> >> Yes thanks for looking into it.  Can you edit
>> testconfiguration.properties
>> >> to add a section for LlapCliDriver tests, see HIVE-7254 for details.
>> That
>> >> way the framework can identify which tests it needs to run and then
>> split
>> >> it into parallel batches across test nodes.  Otherwise it will just
>> run all
>> >> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one
>> huge
>> >> batch.
>> >>
>> >> Once you finish, let me know and I'll tweak the test properties to set
>> a
>> >> batch size for that one.
>> >>
>> >> Thanks
>> >> Szehon
>> >>
>> >> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <
>> prasan...@apache.org
>> >>> wrote:
>> >>
>> >>> Hello everyone
>> >>>
>> >>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split
>> across
>> >>> multiple nodes. I can see from the logs that MiniTezCliDriver are
>> broken
>> >>> down into many tests and ran on different machines.
>> >>>
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>> >>> <
>> >>>
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>> 
>> >>>
>> >>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>> >>> something in PTest2 to enable splitting of tests?
>> >>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>> >>>
>> >>> Thanks
>> >>> Prasanth
>> >
>>
>>
>

Re: Splitting tests in PTest2

[Prasanth Jayachandran]

Szehon,

I committed a different patch now to llap branch, HIVE-11923. This will let 
llap and tez test to run much faster hopefully before the 2h timeout. Once all 
tests passes, committed then we can split up the tests to run in parallel. Even 
after HIVE-11923 if the patch didn’t complete before timeout then I will commit 
the patch and provide you an update to split up the tests.

Thanks
Prasanth

> On Sep 28, 2015, at 7:15 PM, Szehon Ho  wrote:
> 
> OK so there is a bit of a chicken-egg problem, I cannot change it until
> 11642 is in, otherwise it will break the other builds.
> 
> Once its in I can update the properties file to reference the new
> LlapCliDriver.  Maybe you can check it in and I will try enabling it
> tonight.
> 
> On Mon, Sep 28, 2015 at 5:05 PM, Szehon Ho  wrote:
> 
>> OK it may be possible, I will take a look!
>> 
>> Thanks
>> Szehon
>> 
>> On Mon, Sep 28, 2015 at 4:56 PM, Prasanth Jayachandran <
>> pjayachand...@hortonworks.com> wrote:
>> 
>>> @Szehon.. MiniLlapCliDriver  uses the same set of files from minitez. Can
>>> that be reused as such without duplicating?
>>> 
>>> 
 On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran <
>>> pjayachand...@hortonworks.com> wrote:
 
 Thanks Szehon! I will look at it and update you once done.
 
> On Sep 28, 2015, at 6:37 PM, Szehon Ho  wrote:
> 
> Yes thanks for looking into it.  Can you edit
>>> testconfiguration.properties
> to add a section for LlapCliDriver tests, see HIVE-7254 for details.
>>> That
> way the framework can identify which tests it needs to run and then
>>> split
> it into parallel batches across test nodes.  Otherwise it will just
>>> run all
> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one
>>> huge
> batch.
> 
> Once you finish, let me know and I'll tweak the test properties to set
>>> a
> batch size for that one.
> 
> Thanks
> Szehon
> 
> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <
>>> prasan...@apache.org
>> wrote:
> 
>> Hello everyone
>> 
>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split
>>> across
>> multiple nodes. I can see from the logs that MiniTezCliDriver are
>>> broken
>> down into many tests and ran on different machines.
>> 
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>> <
>> 
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>>> 
>> 
>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>> something in PTest2 to enable splitting of tests?
>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>> 
>> Thanks
>> Prasanth
 
>>> 
>>> 
>>