[issue22607] find by dichotomy the failing test

[STINNER Victor]

STINNER Victor added the comment:

> See message 228968 for the rationale:

http://bugs.python.org/issue22588#msg228968

Ok, this use case is now well supported by the new test.bisect tool. Yes, this 
issue is a duplicate of the issue #29512. I forgot this old issue!

--
resolution:  -> duplicate
stage: test needed -> resolved
status: open -> closed
superseder:  -> regrtest refleak: implement bisection feature

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Wasn't the similar feature implemented in issue29512?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Xavier de Gaye]

Xavier de Gaye added the comment:

This new version of the patch uses a specific exception to skip tests and fixes 
a bug when invoking the overriden and wrapped subTest method.

--
Added file: http://bugs.python.org/file36994/regrest_XY_options_2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Xavier de Gaye]

Xavier de Gaye added the comment:

The attached patch adds the '-X' and '-Y' options to the regrtest tool, 
allowing to select a range of tests and a range of their subtests.  The patch 
is missing the test cases for the moment.

Limitation:
Does not work very well with nested subtest (nested subtests are currently only 
used by the unittest test suite itself): subtest numbers are flattened across 
all the nested subtests.

Here is the sequence of commands that would have allowed to find the subtest 
responsible for the crash in test_capi at issue 22588, without modifying any 
code:
./python -m test -X [] test_capi  # Get the test count: 18 
tests.
./python -m test -X range(1,10) -R 23:23 test_capi# pass
./python -m test -X range(10,15) -R 23:23 test_capi   # pass
./python -m test -X range(15,17) -R 23:23 test_capi   # pass
./python -m test -X [17] -R 23:23 test_capi   # pass
./python -m test -X [18] -R 23:23 test_capi   # fail
./python -m test -X [18] -Y [] test_capi# Test 18 has 35 subtests.
./python -m test -X [18] -Y range(1,19) -R 23:23 test_capi  # fail
./python -m test -X [18] -Y range(1,10) -R 23:23 test_capi  # fail
./python -m test -X [18] -Y range(1,6) -R 23:23 test_capi   # pass
./python -m test -X [18] -Y range(6,8) -R 23:23 test_capi   # pass
./python -m test -X [18] -Y [8] -R 23:23 test_capi  # pass
./python -m test -X [18] -Y [9] -R 23:23 test_capi  # fail

Output of the last command:
Test# 18: test__testcapi
subTest# 9: internal, {'name': 'test_incref_decref_API'}

--
Added file: http://bugs.python.org/file36987/regrest_XY_options.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Terry J. Reedy]

Terry J. Reedy added the comment:

I believe the goal, and a better title, is Automate leak discovery within a 
group of tests.  Bisection or dichotomy is a means, not a goal, and should not 
be part of the title.

Leak discovery means 'find a test within the group that has a leak (which we 
know or suspect might be occurring)'. 

'Test' can mean multiple modules, one module, one TestCase class, one test_xxx 
method, one subtest, or one assert.  What might it mean in 'group of tests'?  
Pinning a leak on a particular statement likely has to be done manually; 
eliminate that.  On the other hand, it is feasible (see below) to automatically 
blame a particular test method and perhaps a particular subtest.  So interprete 
'tests' as 'test methods or subtests'.

Unittest can run all test methods in a module, all test methods in a test 
class, or an individual test method.  Idle already has a module browser (Class 
Browser), based on stdlib's pyclbr and Idle's TreeWidget, that could be adapted 
(perhaps using ttk.Treeview instead) to produce a module-test_class-test tree.  
Add buttons to select a particular leak test and others to select manual or 
auto nagivation of the tree and we might have something worthwhile.  It could 
go in Tools/Scripts or PyPI.

I have not used subtests yet since they are not available on 2.7, but I can 
imagine semi-automated editing, saving to a temp file, and running.  I would 
need experience with real subtest bisection to comment on the patch.

--
nosy: +terry.reedy
stage:  - test needed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Xavier de Gaye]

New submission from Xavier de Gaye:

This issue stems from issue 22588.

See message 228968 for the rationale:
Automatize the dichotomy process used to to identify memory leaks, crash, 
reference leak, resource leak, etc. in a failing test.

--
components: Tests
messages: 229067
nosy: haypo, xdegaye
priority: normal
severity: normal
status: open
title: find by dichotomy the failing test
type: enhancement
versions: Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Xavier de Gaye]

Xavier de Gaye added the comment:

See msg 228968 for the rationale.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Xavier de Gaye]

Xavier de Gaye added the comment:

With the attached patch (the patch does reintroduce the bug in 
'test_incref_decref_API' of issue 22588 for testing purposes), it is possible 
to find the failing subtest rapidly:

After identifying the failing test, print the list of subtests in this test and 
their number (35 subsets):
$ export SUBTEST_RANGE=[]
$ ./python -m test -m test__testcapi test_capi

Then run:
$ ./python -m test -m test__testcapi -R 23:23 test_capi

after modifying, each time, the range of subtests to execute, with:
$ export SUBTEST_RANGE=range(1,18)# tests 1-17   result: fail
$ export SUBTEST_RANGE=range(1,9) # tests 1-8result: pass
$ export SUBTEST_RANGE=range(9,13)# tests 9-12   result: fail
$ export SUBTEST_RANGE=range(9,11)# tests 9-10   result: fail
$ export SUBTEST_RANGE=[9]# so it must be test #9, check it now

The strong limitation with this solution is that the subTest context manager 
must now be enclosed in a 'try except unittest.SkipTest' clause and that the 
context manager is used more than 100 times in the test suite.

--
keywords: +patch
Added file: http://bugs.python.org/file36875/subtest_in_range.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Georg Brandl]

Georg Brandl added the comment:

I don't think this feature is generally useful enough to be included.  

* Since you need to modify the test code anyway (adding the try-except), it is 
probably just as much work to do the selection there.

* Why only add selection of subtests, and not of all tests?

In addition, the patch would have to be reworked: eval()ing an environment 
variable is not acceptable.

BTW, I think the commonly used term for this is bisection.

--
nosy: +georg.brandl, michael.foord

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[STINNER Victor]

STINNER Victor added the comment:

I requested the feature because I regulary need bisect (once a month, or
maybe two months).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Georg Brandl]

Georg Brandl added the comment:

 I requested the feature because I regulary need bisect (once a month, or 
 maybe two months).

Always within subtests?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Xavier de Gaye]

Xavier de Gaye added the comment:

 With the attached patch (the patch does reintroduce the bug in 
 'test_incref_decref_API' of issue 22588 for testing purposes)

Sorry for not being more explicit and for being lazy doing a copy paste from 
msg 229022:

* this is not a patch, obviously a patch would not attempt to reintroduce a bug 
(as stated above), this is an experiment that you must 'patch' in order to test 
it (I did mention 'testing purposes' in this messagei, see above)
* this was witten in msg 229022 in answer to Does anyone know how to 
automatize the dichotomy process ? from Victor
* it only deals with subsets as a partial attempt to answer this question 
mostly because this is were most of the time is spent when investigating issue 
22588

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Xavier de Gaye]

Xavier de Gaye added the comment:

@ Georg Brandl
 I don't think this feature is generally useful enough to be included.

 * Since you need to modify the test code anyway (adding the try-except), it 
 is probably just as much work to do the selection there.


You seem to be confusing the feature itself with the implementation.
This feature is generally useful.
The fact that there is an acceptable implementation is another matter (and 
subtest_in_range.diff is not an implementation).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Georg Brandl]

Georg Brandl added the comment:

 You seem to be confusing the feature itself with the implementation.

 The fact that there is an acceptable implementation is another matter
 (and subtest_in_range.diff is not an implementation).

You yourself were calling it a solution.

A feature proposal should either have a description of the new feature, or a 
patch.  Since there is no feature description, I reviewed the patch as being an 
implementation of the desired feature.

 This feature is generally useful.

As described in the issue, I disagree.  If the intended feature is more 
comprehensive, please supply an adequate description or a patch that implements 
it.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22607] find by dichotomy the failing test

[Georg Brandl]

Georg Brandl added the comment:

 Sorry for not being more explicit and for being lazy doing a copy paste from 
 msg 229022:

I see, this was split off another issue (which was already closed).  I agree 
that a bit more verbosity in the initial description would have prevented 
confusion :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com