Re: Standalone GTests

[Ms2ger]

On 05/08/2013 08:06 PM, Benoit Girard wrote:


I personally have a strong preference for keeping tests, particularly unit
tests, running fast. My workflow is to develop new code by testing it via
unit tests so longer turn around times slow down my development workflow.
Sadly we don't have a good a way with dealing with slow but efficient tests
and they end up making it difficult to test locally. But this isn't a
reason to reject a useful test so feel free to check it in. Perhaps we
should consider introducing a concept of smoke tests for developers to run
locally where we can aim to maintain a good trade off between code coverage
and turn around times. But that's orthogonal to your request. I think
splitting GTest into a separate test job should be a prerequisite to not
slow down the build job significantly which AIUI reduces test turn around
times by delaying all test jobs from starting.


Test jobs are kicked off before 'make check' starts.

HTH
Ms2ger
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Standalone GTests

[Benoit Girard]

If you plan on adding more tests to GTest please build against patches in
bug 844288 which will hopefully land soon. Note that I have one outstanding
problem left on Windows where the linking only fails on TBPL jobs.

I personally have a strong preference for keeping tests, particularly unit
tests, running fast. My workflow is to develop new code by testing it via
unit tests so longer turn around times slow down my development workflow.
Sadly we don't have a good a way with dealing with slow but efficient tests
and they end up making it difficult to test locally. But this isn't a
reason to reject a useful test so feel free to check it in. Perhaps we
should consider introducing a concept of smoke tests for developers to run
locally where we can aim to maintain a good trade off between code coverage
and turn around times. But that's orthogonal to your request. I think
splitting GTest into a separate test job should be a prerequisite to not
slow down the build job significantly which AIUI reduces test turn around
times by delaying all test jobs from starting.




On Wed, May 8, 2013 at 1:43 PM, Adam Roach  wrote:

> On 5/8/13 12:10, Gregory Szorc wrote:
>
>> I think this is more a question for sheriffs and people closer to
>> automation. Generally, you need to be cognizant of timeouts enforced by our
>> automation infrastructure and the scorn people will give you for running a
>> test that isn't efficient. But if it is efficient and passes try, you're
>> generally on semi-solid ground.
>>
>
> The issue with the signaling tests is that there are necessarily a lot of
> timers involved, so they'll always take a long time to run. They're pretty
> close to optimally "efficient" inasmuch as there's really no faster way to
> do what they're doing. I suspect you mean "runs in a very short amount of
> time" rather than "efficient."
>
> It should be noted that not running the signaling_unittests on checkin has
> bitten us several times, as we'll go for long period of times with
> regressions that would have been caught if they'd been running (and then
> it's a lot more work to track down where the regression was introduced).
>
> /a
>
> __**_
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/**listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Standalone GTests

[Adam Roach]

On 5/8/13 12:10, Gregory Szorc wrote:
I think this is more a question for sheriffs and people closer to 
automation. Generally, you need to be cognizant of timeouts enforced 
by our automation infrastructure and the scorn people will give you 
for running a test that isn't efficient. But if it is efficient and 
passes try, you're generally on semi-solid ground.


The issue with the signaling tests is that there are necessarily a lot 
of timers involved, so they'll always take a long time to run. They're 
pretty close to optimally "efficient" inasmuch as there's really no 
faster way to do what they're doing. I suspect you mean "runs in a very 
short amount of time" rather than "efficient."


It should be noted that not running the signaling_unittests on checkin 
has bitten us several times, as we'll go for long period of times with 
regressions that would have been caught if they'd been running (and then 
it's a lot more work to track down where the regression was introduced).


/a
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Standalone GTests

[Gregory Szorc]

On 5/8/2013 8:41 AM, Ethan Hugg wrote:

Hi All,

We currently have 10 C++ GTest unittests in Firefox in
media/mtransport/test and media/webrtc/signaling/test that build standalone
tests with the FF build.  These tests currently run as part of the build
and make the B orange if they fail which is why a couple of them are still
stubbed at runtime based on an environment variable.

I am looking into getting them to build on Windows (Bug 862837).  This
appears non-trivial because of the different way the Windows build links.
  It was suggested by Ted that we move them to the new firefox GTest which
would build them only with --enable-gtest and run as part of 'firefox
-unittest'.

So, for those of you interested, my questions are:

1. If they were moved into the standard Gtest, would they still be built by
tbpl and run with the tests instead of as part of the build?


Currently, we have no gtest coverage on checkins. That will change with 
bug 844288.


Initially, we will roll gtest into |make check|, which means they will 
be executed as part of checkin. There is a movement to split out 
long-running components of |make check| into their own automation jobs 
so tests don't always run as part of the build step. See 
https://bugzilla.mozilla.org/show_bug.cgi?id=822394#c3 for part of that. 
But, I don't think the eventual split of |make check| is of concern to 
many at this juncture.



2. One of them in particular (signaling_unittest) is fairly large and takes
minutes to run.  Should it be run separately by using GTEST_FILTER?



I think this is more a question for sheriffs and people closer to 
automation. Generally, you need to be cognizant of timeouts enforced by 
our automation infrastructure and the scorn people will give you for 
running a test that isn't efficient. But if it is efficient and passes 
try, you're generally on semi-solid ground.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform