[issue18103] Create a GUI test framework for Idle

[Terry J. Reedy]

Terry J. Reedy added the comment:

While I have not committed any gui tests, the patch is based on experiments 
with a couple of temporary files.

--
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Roundup Robot]

Roundup Robot added the comment:

New changeset 0767363a0393 by Terry Jan Reedy in branch '2.7':
Issue #18103: Update README.txt and test_idle to describe and run gui tests.
http://hg.python.org/cpython/rev/0767363a0393

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Roundup Robot]

Roundup Robot added the comment:

New changeset c818c215f1a4 by Terry Jan Reedy in branch '3.3':
Issue #18103: Update README.txt and test_idle to describe and run gui tests.
http://hg.python.org/cpython/rev/c818c215f1a4

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Terry J. Reedy]

Terry J. Reedy added the comment:

Must mention that 'support' was 'test_support' in 2.7.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Phil Webster]

Changes by Phil Webster :


--
nosy: +philwebster

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[R. Jayakrishnan]

Changes by R. Jayakrishnan :


--
nosy: +JayKrish

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Terry J. Reedy]

Terry J. Reedy added the comment:

I should have explicitly said 'all test methods in one test file'.

I included ' caller is __main__' in the requires 'pass' condition. My first 
question was only about, for instance, running test_grep from the 'if __name__ 
...' part of GrepDialog, For that, I verified as part of #18081 that setting 
support.use_resources is possible.

My second question was about running all tests in one test_idle file from the 
command line. If not possible, so be it. I just need to document the limitation.

Since multiple skips are acceptable, the 'framework' patch for this issue will 
just be a change to README explaining the use of the gui resource and when it 
is needed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[R. David Murray]

R. David Murray added the comment:

Oh, actually I do know why that is.  It is because requires is called from 
test_main in test_urllibnet, which unittest bypasses.  So if you are calling it 
in particular tests, then I presume that will be an issue for the unittest case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[R. David Murray]

R. David Murray added the comment:

support.requires is smart.  If the caller's module is __main__, it treats the 
resources as set.  So you don't have to do anything special in your __main__ to 
make the tests runnable directly.

The tests are also run if unittest is called.  I don't know why that is, but I 
tested it on test_urllibnet, which is protected by the 'net' resource, and the 
tests ran.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Nick Coghlan]

Nick Coghlan added the comment:

Indeed, I think the most sensible course here is to require the use of regrtest 
to run the GUI tests. Our other resource-dependent tests all fall into that 
category already.

If the stdlib unittest provided a resource system, we'd rewrite the regrtest 
resource system to be based on that instead, but that's not currently the case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Ned Deily]

Ned Deily added the comment:

Why not just run the tests under regrtest like all other standard library 
tests?  From the build directory, you could use something like (Windows may be 
a bit different):

./python.exe -m test -h  # to get regrtest options help
./python.exe -m test -ugui -v test_idle  # to run just the idle tests with 
verbose on

For 2.7, you'll need to use '-m test.regrtest'.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Terry J. Reedy]

Terry J. Reedy added the comment:

Thanks. That still leaves the problem of getting all tests to run during 
development when running them with unittest. Calling requires('gui') passes 
when (_is_gui_available() and (caller is __main__ or 'gui' in use_resources)). 
Use_resources is normally set by regrtest. As far as I saw, most other support 
functions work fine with unittest and are not dependent on regrtest.

The doc for unittest.main say that the argv argument can be a list of options 
passed to the program. Which program? Can that be used to set a resource?

I plan to try
from test import support; support.use_resources = ['gui']
in the 'if __name__ ...' sections of xyz.py and test_idle.py, but that will not 
work from the command line ('python -m unittest whatever').

An alternative is to have requires() check whether the test is running under 
regrtest (which supposedly changes support.verbose from 1 (which it starts as) 
to 0) and passing, at least for 'gui', if not. (Is there any other way for a 
test to detect the test runner?)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Ned Deily]

Ned Deily added the comment:

Individual test cases are only seen if you run regrtest (-m test) in verbose 
mode (-v).  There are many test cases that are akipped in the standard library, 
depending on platform and -u options, and you normally don't see them (without 
-v).  So mixing gui-dependent test cases in with others shouldn't be a problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Terry J. Reedy]

Terry J. Reedy added the comment:

The basic question for this issue is whether to segregate gui tests in a 
separate directory (idle_test_gui?) and run them with a separate 
test_idle_gui.py? or to sprinkle gui test cases throughout the test suite, 
perhaps one to each test_idle/text_xxx.py file?

For development, I am pretty sure I would prefer the latter. I have started 
test_grep for the 3 methods that can be gui free, but if (when) I do a 
gui-requiring test of the dialog itself, I would prefer to have it follow in 
the same file.

However, my impression is that each requires('gui') call will result in a skip 
message on the test output. (Correct?) So my question is: will non-idle 
developers tolerate over 50 skip warnings from Idle tests? or is there a way to 
suppress multiple warnings and consolidate them into just one if there is at 
least one?

--
nosy: +ncoghlan, r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Terry J. Reedy]

Terry J. Reedy added the comment:

Thanks Ned. I commented on the issue. The code in the proposed function will 
fix one of the two problems with tkinter.Variable that I noted in #18131. I can 
imagine there might be situations in which it would also be useful for testing, 
as GP claims.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Ned Deily]

Ned Deily added the comment:

Terry: FYI, I just noticed the languishing patches in Issue4343 which *might* 
be of some use here.

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Terry J. Reedy]

Terry J. Reedy added the comment:

My experiments and some web postings indicate that if a tkinter class has a 
master or parent option, it may not really be an option, regardless of what our 
docs imply. If tkinter.Tk is called either directly or indirectly, the graphics 
system is initiated and something is displayed. This appears to includes the 
seemingly non-graphics Variable (Var) classes such as IntVar (#18131). I 
started idle_lib/mock_tk.py as part of #18130. This will allow non-gui unit 
testing of any method whose tkinter use is limited to Vars and message boxes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Todd Rovito]

Changes by Todd Rovito :


--
nosy: +Todd.Rovito

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Terry J. Reedy]

Terry J. Reedy added the comment:

I opened #18104 for the human-required gui tests described in the last 
paragraph above.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Ezio Melotti]

Changes by Ezio Melotti :


--
nosy: +ezio.melotti

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18103] Create a GUI test framework for Idle

[Terry J. Reedy]

New submission from Terry J. Reedy:

This is a follow-up to #15392, which created the Idle test framework. The 
similar endpoint for this issue will be a framework for tests that require a 
gui resource, at least one prototype test file, and documentation of Idle 
practice. At the moment, completely automated tests that can run on buildbots 
with a gui, but without human intervention, are a lower priority to me than 
adding text-only tests.

The tkinter test framework should be a starting point, but some things should 
be modernized, as they were for the basic framework. This means using unittest 
rather than regrtest. One thing to copy may be to put gui-tests in a separate 
directory. I presume load_tests() could conditionally load tests in such a 
directory.

It is not clear to me to what extent tk widgets can be created and manipulated 
without being shown on a screen or needing a 'gui' resource.

Idle tests are not intended to test tkinter and tk. A mock_tk could replace at 
least some gui use. Creating a mock-tk framework would be a different issue 
from this one. A generic tk mock might possibly go in a tkinter subdirectory. 
Mocks of Idle classes would belong somewhere below idle_test.

As noted in #15392, about 20 idlelib files have primitive non-automated 'main' 
tests requiring unspecified human action and observation, some of which do not 
even run. The 'test' is to see whether the action produced the expected 
response (which one has to know) or to check the appearance of a text or dialog 
widget. A first step might be to upgrade and semi-automate these instructions 
with unittests in idle_test/human. Running them all with one documented command 
or import would be much nicer pleasant than having to discover and run each 
individually. I might make this a separate sub-issue.

--
assignee: terry.reedy
messages: 190381
nosy: terry.reedy
priority: normal
severity: normal
stage: needs patch
status: open
title: Create a GUI test framework for Idle
type: enhancement
versions: Python 2.7, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com