[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Change by Terry J. Reedy :


--
components: +IDLE

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Roundup Robot]

Roundup Robot added the comment:

New changeset 0a6d51ccff54 by Terry Jan Reedy in branch '2.7':
Issue #18104: revise docstrings, remove obsolete comments.
http://hg.python.org/cpython/rev/0a6d51ccff54

New changeset 6d2982ff441f by Terry Jan Reedy in branch '3.4':
Issue #18104: revise docstrings, remove obsolete comments.
http://hg.python.org/cpython/rev/6d2982ff441f

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Terry J. Reedy added the comment:

The revised patch, as pushed, will serve as a base for this portion of 
Saimadhav's GSOC project. #21477 proposes changes to htest itself. Adding tests 
from other modules will be dependencies of that issue.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
Added file: http://bugs.python.org/file35222/htest5.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Roundup Robot]

Roundup Robot added the comment:

New changeset aea4f427902f by Terry Jan Reedy in branch '3.4':
Issue #18104: News for 3.4 (which will not merge forward).
http://hg.python.org/cpython/rev/aea4f427902f

New changeset b7bc43e96041 by Terry Jan Reedy in branch 'default':
#18104: null merge of 3.4 News entry
http://hg.python.org/cpython/rev/b7bc43e96041

New changeset 7cdb38fa191e by Terry Jan Reedy in branch 'default':
Issue #18104: News for 3.4, in proper place.
http://hg.python.org/cpython/rev/7cdb38fa191e

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Roundup Robot]

Roundup Robot added the comment:

New changeset 460203eaf731 by Terry Jan Reedy in branch '2.7':
Issue #18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
http://hg.python.org/cpython/rev/460203eaf731

New changeset 617656f7b538 by Terry Jan Reedy in branch '3.4':
Issue #18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
http://hg.python.org/cpython/rev/617656f7b538

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Terry J. Reedy added the comment:

Here are my notes on modules that should probably be included in htest. Most 
have some end-of-module test now, many with errors.

--
Added file: http://bugs.python.org/file35119/18104-htest.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Terry J. Reedy added the comment:

Defective patches removed and replaced with one that includes current htest.py 
and should have proper (unix) line endings.

--
Added file: http://bugs.python.org/file35116/18104-htest4.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Changes by Terry J. Reedy :


Removed file: http://bugs.python.org/file35099/18104-htest3.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Changes by Terry J. Reedy :


Removed file: http://bugs.python.org/file35096/18104-htest2.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Terry J. Reedy added the comment:

New patch. I refined the definition of the parameter for run() in the process 
of adding a test for EditorWindow.HelpDialog. This cannot be tested directly. 
This example is also a reminder that some files define more than one display 
widget.

I should try template and framework with non-dialog, but the example above 
pretty well convinces me that the framework should be flexible enough.

--
Added file: http://bugs.python.org/file35099/18104-htest3.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Terry J. Reedy added the comment:

The new patch adds a docstring with spec template, a second example, and a 
crude runall(). I am inclined to push this as a base for further patches -- at 
least one to improve runall and others to test more widget classes. These might 
be separate issues.

--
Added file: http://bugs.python.org/file35096/18104-htest2.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Terry J. Reedy added the comment:

After correcting bugs in the 'explaining with code' posted before, and making a 
few other changes, the attached 3.3 diff works. The main substantive change is 
that I added a parameter to the class to move the tested dialog under the 
toplevel driver so the test message would be visible. The default for this 
dialog is centered over the parent (normally the config dialog).

If I still like this design after adding a couple more tests and a run_all 
function, I  will commit it.

--
keywords: +patch
stage: needs patch -> patch review
Added file: http://bugs.python.org/file34410/18104-htest1.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Terry J. Reedy added the comment:

I thought through my design criteria a bit, and in the process, decided on a 
specific concrete proposal to test.

1. It should be possible to run one test. It should easy to run (or not) the 
test (or tests) for a module when editing its file. It should be easy to tell 
when editing that there is a human test. These would both be served the a 
couple of lines in the 'if name' block (after the unittest lines) such as

from idlelib.idle_test.htest import run
run(GetCfgSectionNameDialog)  # or other class

2. It should be possible to see which modules have a human test. But I do not 
want a directory of 20+ 10-line files.

3. I want to factor out as much boilerplate code as possible. This will both 
make it easier to add tests and to modify the behavior of all tests at once.

Something like the following may work for both criteria.

In idle_test/htest.py

GetCfgSectionNameDialog_spec = {
  'kwds': {'title':'Get Name',
 'message':'Enter something',
 'used-names': {'abc'}},
  'msg': "After the text entered with [Ok] is stripped, , "
 "'abc', or more that 30 chars are errors. "
 "Close with a valid entry (printed), [Cancel], or [X]"}

def run(klas):
  root = tk.Tk()
  klas_spec = globals[klas.__name__+'_arg']
  klas_spec.kwds['parent'] = root # does Idle use 'parent' consistently?
  def run_klas():
widget = klas(**klas_spec.kwds)
try:
  print(widget.result)
except AttributeError:
  pass
  Message(root, text=klas_spec.msg).pack()
  Button(root, text='Test ' + klas.__name__, command=run_klas).pack()
  root.mainloop()

4. It should be possible to discover and run all tests. With one object per 
widget in htest, each with a marked name, it is easy to filter globals for 
marked objects.

  filter(lambda ob: hasattr(ob, '__name__') and '_args' in ob.__names__, 
globals())

--
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Terry J. Reedy added the comment:

Some follow-up after revising the test for GetCfgSectionNameDialog in  
configSectionNameDialog.py (#18130):

1. I am no longer concerned about automated discovery. Anyone adding a widget 
can just as easily add something to a master list as add a file to be 
discovered. I meant to type up and post my handwritten list of files with 
widget tests, with the status of each. I will try to find it or recreate it.

2. I am not so much concerned about an automated summary. Once the tests work 
correctly, errors due to editing the underlying code should be caught by 
immediate testing.  The model I used is

if __name__...:
  
  

where  currently means to open a tk window with a button to 
start the test, but which can instead just be closed [x] to skip the test. My 
idea is that after editing, one should always run the automated test (if there 
is one, which there eventually will be) and maybe run the visual test.

3. For #18130, I left the test code in the 'if name' clause. To run all such 
tests from a separate master htest file, it needs to be moved into a function. 
I am thinking of '_htest'. (Some files already have the visual test code 
pulling into a 'main' function, but to me that is too similar to 'unitest.main' 
and similar.) Or perhaps the code should be moved to an idle_test/htest.py 
file, but with a means to run just the one test from the module file.

4. For this test, the widget has a message parameter, which I used for test 
hints. Other widgets do not have such, and the test hints will have to be in 
the driver window.

I need to pick, say, 3 tests and do some experiments.

--
nosy: +sahutd

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[R. Jayakrishnan]

Changes by R. Jayakrishnan :


--
nosy: +JayKrish

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Terry J. Reedy added the comment:

I opened #18130 for an updated configSectionNameDialog.py patch. It also adds 
to idle_tests two files, mock_tk.py and test_config_name.py. The latter uses 
the former for gui-free automated tests of some of the dialog methods. I plan 
to commit in a couple of days. The fixed human test is required for this issue.

--
dependencies: +idlelib.configSectionNameDialog: fix and add tests and mocks

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Changes by Terry J. Reedy :


Removed file: http://bugs.python.org/file30439/configSectionNameDialog.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Ezio Melotti]

Changes by Ezio Melotti :


--
nosy: +ezio.melotti

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Terry J. Reedy added the comment:

While improving the self-test for confixSectionNameDialog.py after fixing the 
error that stopped it from running, I discovered an error in the NameOk method: 
names were not really stripped before being tested. so duplicate section names 
were erroneously accepted if initially surrounded by whitespace. After fixing 
that, I removed unneeded code, including 5 pairs of parentheses in one 
statement, and added recommended spaces. I also centered the action buttons in 
the dialog.

Attached is a drop-in replacement for the file (produced with Idle on my 
non-developement  Win7 machine, so probably with the extra line end char). I 
think it is good enough to commit, but I am curious if the instructions added 
to the test are clear enough. (The dialog is the one seen if Save as New Custom 
Theme (Highlighting tab) or Save as New Custom Key Set (Key tab) are selected 
on the Options / Preferences dialog.

For new custom test files, the root window should be open farther from the edge 
of the screen.

--
stage: needs patch -> patch review
Added file: http://bugs.python.org/file30439/configSectionNameDialog.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

Terry J. Reedy added the comment:

What I want is for the tests to be discovered when they should be and not when 
they should not ;-) -- without jumping through too many hoops. I probably once 
read 'recursing into subdirectories' but forgot. I like 'h_test...' for 'human 
test ...'.

Thinking about it more, unittest does not not add much to this type of test. If 
dialogs maps dialog to test information used by h_test_dialog, then a first 
draft of testing all (there are 6 that I know of, 3 with display tests) could 
amount to
def h_test_dialogs():
  for dialog in dialogs:
h_test_dialog(dialog)
New entries to dialogs would be 'discovered' as added.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Todd Rovito]

Changes by Todd Rovito :


--
nosy: +Todd.Rovito

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[R. David Murray]

R. David Murray added the comment:

Since you *don't* want these to be autodiscovered by testing frameworks, 
perhaps h_test_.py would be a better naming scheme.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18104] Idle: make human-mediated GUI tests usable

[Terry J. Reedy]

New submission from Terry J. Reedy:

23 of the 62 idlelib/*.py files have an 'if __name__ ...' test that brings up a 
tkinter gui widget and waits for the human tester to do ... something. Problems:

0. They are a bit of a nuisance to run individually: either type
python -m idlelib.SomeLongNameYoumightMistype
or open in an editor and hit F5.

Running all is much worse given that I found no list; doing the above with all 
62 files to find the 23 is something few would do.

1. About 5 do not run; at least 1 is a simple incomplete 2to3 fix (which means 
it has never been run with 3.x). I will post separately any that are not 
obvious.

2. They were written by and for the module authors; some are a puzzle. For 
example: python -m idlelib.FileList. I do not know what to do to perform the 
test.

3. About 4 of those that do run do not stop properly, depending on how run. 
Example: after python -m idlelib.ColorDelegator, the puzzling window (blank on 
3.3, not 2.7) seems to close correctly, but open the same file and run with F5 
(2.7 or 3.3, Windows) and pressing [x] in the window does nothing. But most do 
close so this is something different with a few.

4. Even when the test is fairly obvious, (python -m idlelib.MultiStatusBar) 
there is no way to indicate failure so that an  automated record of failures is 
collected.

My overall idea expands on the example of the dialog tests, such as
python -m idlelib.aboutDialog
Each opens a master test window with a start button that opens a window with 
the dialog be tested. Make a standardized master window for all tests, add an 
explanation of what to do (or a button to display it) and buttons to record the 
result: OK, NothingHappened, or Error (explain in text widget). A test method 
would set up the explanation and test item, connect both to a master window, 
run, retrieve result (I have no idea how to do this ;-), and end with 
AssertTrue(result, explanation).

I am thinking to suffix human-only tests with '_H' and put 'test_xyy_H' in a 
subdirectory such as idle_test/human. Some test files might test several 
different modules, such as 'test_dialog_H'.

--
assignee: terry.reedy
messages: 190388
nosy: roger.serwy, terry.reedy
priority: normal
severity: normal
stage: needs patch
status: open
title: Idle: make human-mediated GUI tests usable
type: behavior
versions: Python 2.7, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com