-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 23/03/11 14:53, Barry Warsaw wrote:
> If 'make quicktest' were actually quick - say could run in 1/10 the current
> time, it could be used as a smoke test for merge-dance cases. OTOH, running
> some localized test for the feature or bug you're tryi
Nick Coghlan writes:
> (i.e. less than a minute) because you're going to be running it three
> times in quick succession (perhaps 3 if it applies to 2.7 as well).
Nobody says it's got to be *you* that runs the tests, just that they
need to be run before pushing to public repo.
Here's a simple
On Thu, Mar 24, 2011 at 7:07 AM, Georg Brandl wrote:
> On 23.03.2011 18:10, Barry Warsaw wrote:
>> On Mar 23, 2011, at 05:09 PM, Antoine Pitrou wrote:
>>
>>>That's completely bogus. There's no reason to believe that a push race would
>>>favour certain regressions over certain others. Again, you ne
On 23.03.2011 18:10, Barry Warsaw wrote:
> On Mar 23, 2011, at 05:09 PM, Antoine Pitrou wrote:
>
>>That's completely bogus. There's no reason to believe that a push race would
>>favour certain regressions over certain others. Again, you need the full test
>>suite to assert that no regressions occu
s...@pobox.com wrote:
Barry> Not if you keep in mind the appropriate use case for each of the
Barry> separate make test targets.
Programmers are lazy. They will often take the shortest path. Fix a small
bug in module X which seems innocent enough, fail to recognize that it
breaks modul
On Mar 23, 2011, at 05:09 PM, Antoine Pitrou wrote:
>That's completely bogus. There's no reason to believe that a push race would
>favour certain regressions over certain others. Again, you need the full test
>suite to assert that no regressions occured. (or you might as well run 10
>tests at ran
Barry> Not if you keep in mind the appropriate use case for each of the
Barry> separate make test targets.
Programmers are lazy. They will often take the shortest path. Fix a small
bug in module X which seems innocent enough, fail to recognize that it
breaks module Y. Run "make smokete
Barry> If I disable down to and including test_concurrent_futures,
Barry> quicktest runs in 3m20s wall clock. *That's* what I'm talkin'
Barry> 'bout!
How do you know you didn't eliminate the most important tests? (That is,
the stuff which tests the code which is currently the most f
On Wed, 23 Mar 2011 11:18:50 -0400
Barry Warsaw wrote:
>
> This is just the opposite. I'm not saying people shouldn't run the full(-ish)
> test suite before committing, I'm saying we should have a really fast minimal
> set of tests as a smoke test when dealing with push-races.
That's completely
On Mar 23, 2011, at 10:02 AM, s...@pobox.com wrote:
>Eliminating tests based on the time it takes to run them suggests that the
>resulting smaller test run may have considerably different overall coverage
>quality than you might desire. Some tests (syntax, basic arithmetic, etc)
>probably run bla
On Wed, 23 Mar 2011 10:44:30 -0400
Barry Warsaw wrote:
> In IRC Antoine suggested -j5 (note that -j is not compatible with -l so you
> have to override TESTOPTS not just EXTRATESTOPTS). Adding --slow here's what
> I get:
>
> $ make TESTOPTS="-j5 --slow" quicktest
> ...
> 10 slowest tests:
> test
On Mar 24, 2011, at 12:29 AM, Nick Coghlan wrote:
>Entirely independent of the "make quicktest" question, it would be
>nice if the default behaviour of regrtest was updated to check the
>number of cores a machine has and default to using that many processes
>(leaving people to turn it down if they
On Mar 23, 2011, at 03:08 PM, Antoine Pitrou wrote:
>On Wed, 23 Mar 2011 09:53:37 -0400
>Barry Warsaw wrote:
>> OTOH, running
>> some localized test for the feature or bug you're trying to land might be
>> enough.
>
>Might indeed. Quite often, though, some change in a library affects
>another one
Antoine> Does anyone use "make quicktest" for something useful?
I don't use it at all.
Antoine> There is a reason the regression test suite has many tests...
Antoine> "Blacklisting" some of them sounds like a bad thing to do.
+1.
Eliminating tests based on the time it takes to run
In IRC Antoine suggested -j5 (note that -j is not compatible with -l so you
have to override TESTOPTS not just EXTRATESTOPTS). Adding --slow here's what
I get:
$ make TESTOPTS="-j5 --slow" quicktest
...
10 slowest tests:
test_mmap: 221.6s
test_shelve: 184.4s
test_posix: 156.3s
test_largefile: 150
On Thu, Mar 24, 2011 at 12:08 AM, Antoine Pitrou wrote:
> Really, people already don't run the test suite enough before
> committing/pushing (and ironically these same people often don't check
> the buildbots afterwards). I don't think we want to promote more laxism.
Encouraging a step up from "n
On Wed, Mar 23, 2011 at 11:53 PM, Barry Warsaw wrote:
> In any case 'make quicktest' isn't really being honest with us . We
> should fix it or remove it.
It took about 11 minutes wall clock time for me. One thing I noticed
is that it does the "run it twice to ensure the .pyc files are there
the
On Wed, 23 Mar 2011 09:53:37 -0400
Barry Warsaw wrote:
> OTOH, running
> some localized test for the feature or bug you're trying to land might be
> enough.
Might indeed. Quite often, though, some change in a library affects
another one (especially when we're talking about things like socket
or t
On Mar 23, 2011, at 02:31 PM, Antoine Pitrou wrote:
>Does anyone use "make quicktest" for something useful?
Not currently. Can it be made useful? Should it be removed?
>There is a reason the regression test suite has many tests...
>"Blacklisting" some of them sounds like a bad thing to do.
If
On Wed, Mar 23, 2011 at 11:31 PM, Antoine Pitrou wrote:
> On Wed, 23 Mar 2011 22:49:39 +1000
> Nick Coghlan wrote:
>> On Wed, Mar 23, 2011 at 9:45 PM, John Arbash Meinel
>> wrote:
>> > I don't specifically know what is in those 340 tests, but 18min/340 =
>> > 3.2s for each test. Which is *much*
On Wed, 23 Mar 2011 22:49:39 +1000
Nick Coghlan wrote:
> On Wed, Mar 23, 2011 at 9:45 PM, John Arbash Meinel
> wrote:
> > I don't specifically know what is in those 340 tests, but 18min/340 =
> > 3.2s for each test. Which is *much* longer than simple smoke tests would
> > have to be.
>
> The cou
On Wed, Mar 23, 2011 at 9:45 PM, John Arbash Meinel
wrote:
> I don't specifically know what is in those 340 tests, but 18min/340 =
> 3.2s for each test. Which is *much* longer than simple smoke tests would
> have to be.
The counts Barry is referring to there are actually counting test
*files*, ra
22 matches
Mail list logo
