[issue47133] enhance unittest to show test name and docstring on one line

[Gregory P. Smith]

Gregory P. Smith  added the comment:

A more fundamental issue behind this: unittest's plain text verbose output is 
not intended to be parsable and consumable by machines. It is 22 years old and 
was intended for use in 80 column wide terminals as a progress report while 
watching it run as much it is a final result.  Consider that the legacy use 
case or format if you wish.

Allowing an option for output to one line per test with description could be 
more likely to require horizontal scrolling or reading past a line wrap in CI 
setups where text is scrolling regions embedded in scrolling regions.  Yuck.  
BUT not necessarily bad.  Just different.

The ideas expressed in this thread of making a more clear text format with the 
important PASS/FAIL/ERROR/SKIP status up front and a very clear pastable test 
module.class.test_name seem wise.

I expect most Python projects with nicer test output formatting desires have 
simply adopted pytest and never looked back.

---

Other python unittest derivative concepts to get clear output of results:

We add a reliably machine parsable XML report of test outcomes in absl-py 
https://github.com/abseil/abseil-py/tree/main/absl/testing. It's IMNSHO a 
rather annoying implementation - we wish unittest made plugging in additional 
output formats easier. We use that generated XML format for results from 
unittest systems in all of our languages at work, it is picked up and parsed by 
our internal CI system and used to display nicely formatted results when 
available.

I assume pytest also provides machine parsable reporting capabilities but I 
haven't looked.

--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47133] enhance unittest to show test name and docstring on one line

[Steven D'Aprano]

Steven D'Aprano  added the comment:

Rather than an option for this, it would be cleaner to deprecate the 
current output in 3.11, and fix it in 3.12 or 3.13.

Otherwise we have to maintain the old (bad?) output and the new output 
both forever.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47133] enhance unittest to show test name and docstring on one line

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

This is already the third (at least) issue asking for such feature. Many people 
have problems with the current output. I would prefer to have a full qualified 
name of the test which I can copy by a double click and insert in a command 
that reruns specified tests. And full path to the test's source which I can 
copy to run in editor. pytest output looks more user-friendly to me.

But it may break other's test output parsers so we perhaps need an option for 
this. Options for unittest and libregrtest. I am not sure that it is a task for 
an average new contributor.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47133] enhance unittest to show test name and docstring on one line

[Steven D'Aprano]

Steven D'Aprano  added the comment:

I have no problem with pinging Python-Dev, or any other forum, asking 
for interested parties. I just don't think we should be intentionally 
spliting the discussion more than it's going to get split organically. 
(If this is interesting to people, they will discuss it on other forums 
no matter what we say or do.)

Aside from the conservative position "If it ain't broke, don't break 
it", also known as "Chesterton's Fence":

https://matt-rickard.com/chestertons-fence/

I'm not specifically opposed to this change. Nor am I in favour.

Why do we print the first line of the doctests in the output of failing 
tests? If we understood the rationale for why we do that, it might give 
some insight into the consequences of removing it. If we don't know why 
it was shown in the first place, maybe we shouldn't remove it until we 
have some idea.

I am -1 on another option flag unless there is no other choice. Each 
boolean flag option doubles the number of tests of unitttest itself 
needed for full coverage.

Another option might be to leave the docstring line alone, and just 
*add* the 'ERROR' to the method name line:

# current output
test_broken_doc (mathlib.testing.test_utils.TestMinmax)
This is a docstring ... ERROR

# enhancement
test_broken_doc (mathlib.testing.test_utils.TestMinmax) ERROR
This is a docstring ... ERROR

That seems to me to be less risky than trying to cram them all on one line.

test_broken_doc (mathlib.testing.test_utils.TestMinmax) This is a docstring 
... ERROR

By the way, I presume we do the same thing for FAIL lines, which have 
the same behaviour as ERROR lines:

test_fail_doc (mathlib.testing.test_utils.TestMinmax)
This is a docstring ... FAIL

test_fail_nodoc (mathlib.testing.test_utils.TestMinmax) ... FAIL

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47133] enhance unittest to show test name and docstring on one line

[Ethan Furman]

Ethan Furman  added the comment:

I proposed a discussion to python-dev to increase the odds that folks with an 
interest could chime in.  Not everyone follows the new-bugs list.

I find having the output on two lines counter-intuitive -- I was expecting one 
line per test, and indeed my experience seemed to support that (out of hundreds 
of tests showing on one line, it's easy to miss that the rare verbose error is 
on two).  And I'm not the only one, since another developer also thought there 
was a problem and spent most of issue46126 "fixing" it.

Another reason for posting to python-dev is that issue 46126 had been active 
for three months and had a PR merged before another dev mentioned that verbose 
tests with docstrings just printed on two lines instead of one.

Thank you for pointing out the possible disruption caused by unittest output 
changes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47133] enhance unittest to show test name and docstring on one line

[Steven D'Aprano]

Steven D'Aprano  added the comment:

I don't think this change of behaviour should be accepted without discussion, 
and I'm not sure why you think Python-Dev is the right place, rather than here. 

Messing around with unittest and docstrings has already caused issues in the 
past:

https://docs.python.org/3/library/unittest.html#unittest.TestCase.shortDescription

so perhaps we shouldn't break what's not broken?


And frankly, I don't see why the current behaviour is a problem, although maybe 
that's because the way I use unittest is different from the way you use it. 
Grepping the output for errors? Why run verbose mode if you aren't going to 
read the whole output? Seems very odd.

Ethan says: "the test name is easily missed by anyone who isn't aware of that"

Surely the solution to that is education, not changing unittest? Now you are 
aware of it, and the test name is no longer "easily missed".

If it ever was. To me, it is plain as day. Here is some typical output:


[steve ~]$ python3.10 -m unittest -v mathlib/testing/test_utils.py
test_broken_doc (mathlib.testing.test_utils.TestMinmax)
This is a docstring ... ERROR
test_broken_nodoc (mathlib.testing.test_utils.TestMinmax) ... ERROR
test_empty_iterable (mathlib.testing.test_utils.TestMinmax) ... ok
[additional passing tests ommitted for brevity]

==
ERROR: test_broken_doc (mathlib.testing.test_utils.TestMinmax)
This is a docstring
--
Traceback (most recent call last):
  File "/home/steve/Desktop/mathlib/testing/test_utils.py", line 20, in 
test_broken_doc
self.assertEqual(1, x)
NameError: name 'x' is not defined

==
ERROR: test_broken_nodoc (mathlib.testing.test_utils.TestMinmax)
--
Traceback (most recent call last):
  File "/home/steve/Desktop/mathlib/testing/test_utils.py", line 24, in 
test_broken_nodoc
self.assertEqual(1, x)
NameError: name 'x' is not defined

--
Ran 10 tests in 0.004s

FAILED (errors=2)


Errors are listed twice, once in the summary, and the second time in the 
details section showing the traceback. Note that if you just grep for ERROR you 
get the result you want:

[steve ~]$ python3.10 -m unittest -v mathlib/testing/test_utils.py |& grep ERROR
This is a docstring ... ERROR
test_broken_nodoc (mathlib.testing.test_utils.TestMinmax) ... ERROR
ERROR: test_broken_doc (mathlib.testing.test_utils.TestMinmax)
ERROR: test_broken_nodoc (mathlib.testing.test_utils.TestMinmax)


So I am having trouble seeing what the problem here is.

--
nosy: +steven.daprano

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47133] enhance unittest to show test name and docstring on one line

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Issue14265, issue46126.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47133] enhance unittest to show test name and docstring on one line

[Itay Yeshaya]

Itay Yeshaya  added the comment:

I would like to work on this.

--
nosy: +itay.yeshaya

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47133] enhance unittest to show test name and docstring on one line

[Irit Katriel]

Change by Irit Katriel :


--
keywords: +easy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47133] enhance unittest to show test name and docstring on one line

[Ethan Furman]

Change by Ethan Furman :


--
assignee:  -> ethan.furman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47133] enhance unittest to show test name and docstring on one line

[Ethan Furman]

New submission from Ethan Furman :

When running unittest with the -v flag, if a test has errors, and has a 
docstring, the test name is shown on one line, and the docstring is shown on 
the next line -- and the ERROR word is shown with the docstring.

What this means is that:
- the test name is easily missed by anyone who isn't aware of that (I only 
recently found out when someone else pointed it out to me)
- grepping for ERROR doesn't reveal the test name, necessitating a search 
through the testing code to find the test that failed

There are two possible solutions, and the selection of which one should be 
discussed on python-dev:
- print the name and docstring, and ERROR, all on one line; or
- provide a command-line switch to select that behavior

---

For the experienced developers who see this:  please leave the issue for one of 
the new developers subscribed to core-mentorship.  Thank you.

--
components: Tests
messages: 416090
nosy: ethan.furman
priority: low
severity: normal
stage: needs patch
status: open
title: enhance unittest to show test name and docstring on one line
type: enhancement
versions: Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com