[Python-ideas] Re: PEP Draft: Power Assertion
On Fri, Sep 24, 2021 at 8:51 PM Steven D'Aprano wrote: > > Pytest needs to take heroic measures in an import hook to re-write your > assertions to do what it does: > > Maybe a language change, with a public API for assertions, would make it > much easier and less hacky for test frameworks to do what Pytest does. > Now that's a PEP I could get behind. That's what I was getting at with asking if there were language changes that would be required to really do this right. Adding one feature to unittest doesn't seem worth it to me -- adding something to the language that makes it possible (easier) for all test frameworks to be featureful would be a fine idea. I'd love to hear what the pytest folks think of this. It wouldn't be the first time that Python added features to the language > to make it easier for third-party libraries. > agreed. > Also, Pytest only supports CPython and PyPy. > > https://pytest.org/en/6.2.x/getting-started.html Well, if the other implementations want a good test framework, that's up to them. it would be nice if better assertions was a language feature rather than > requiring heroic import hook and AST hacks to work. I suspect that the "heroic measures" pytest takes are the challenge for other implementations, but so would adding this kind of functionality to all implementations -- putting it in official Python would be essentially requiring other implementations to do so -- but the developers of those would still need to do the work. (that being said, one option I suggested what that a shiny new featureful test framework be added to the stdlib :-) ) > 2) there are a lot of things in Pytest that are very useful. > > Will they be less useful if Python gives better diagnostics on assertion > failures? > Of course not -- my point was that if you want a better test framework, you will want more than just this one feature -- and it you're going to use pytest (or something else) anyway, then why do you need this in the stdlib. > And a lot of problems with unittest. > Such as? > That's a long story that I've been meaning to write up for ages, but I'll make these few points: 1) Many major (and minor) projects use pytest, even though unittest is in the stdlib -- why is that? And why did they even bother writing pytest (and nose, back in the day) 2) unitest is not very open to extending -- no one wants to write many more assertions, and yet, we then don't have many. 3) This is, I suppose, personal taste, but I never liked unitest, it's just too awkward and verbose for me -- the second I discovered pytest I never looked back. And I'm not alone in that. The only really unpleasant part of unittest is having to remember the > spellings of all the various `self.assertSpam...` methods. Otherwise, > it's a pretty nice framework. > The way to test Exception raising (assertRaises ?) is painful :-( But I know you have looked at the self.assertSpam methods -- why have them at all? all they do is provide a nice failure message (at least the ones I've looked at) with pytest, you don't need them at all. And there are any number of asserts that don't exist, so you have to use assertTrue, and then you don't get any benefit at all. If assertions were better, perhaps we could fix that. > yup -- then you wouldn't need any assertSpam methods -- then we could get rid of the TestCase classes, and you'd have something like pytest ;-) -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/M5YPKG3ZUGENBWEQP5N73XAXMHV5IJIB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP Draft: Power Assertion
Guido van Rossum writes: > I think this is by far the best option. Pytest can evolve much faster than > the stdlib. Is there no room for making it easier to do this with less invasive changes to the stdlib, or are Steven d'A's "heroic measures in an import hook" the right way to go? Other Steve ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FIEHJ6EXUJ25USF7VW3LF7BJQNNV4OAT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP Draft: Power Assertion
This proposal sounds very promising. Some observations/questions follow.
On Fri, Sep 24, 2021 at 02:04:21PM +0300, Noam Tenne wrote:
> Assertion failed:
>
> sc.val['d'] == f
> | ||
> | eFalse
> |
> {'d': 'e'}
Can we see what the output would look like in larger, more complex
examples? Especially those that cross line boundaries?
strict = False
direction = 'left'
sym = 'N'
count = 2
clusters = [None]
assert ((strict and direction is not None)
or (sym == 'N' and count == 1 and len(clusters) == 1)
), "description ..."
How about examples where one of the values is extremely large?
sc.val = {'d': 'e'}
sc.val.update(dict.fromkeys(range(100))
assert sc.val['d'] == 'f'
If the handler is a hook in sys, like displayhook and except hook, then
people can install their own handlers. But I think the default handler
(asserthook?) should do the right thing for ugly assertions.
> In the python-ideas mailing list, Serhiy Storchaka suggests:
>
> It needs a support in the compiler. The condition expression should be
> compiled to keep all immediate results of subexpressions on the stack.
> If the final result is true, immediate results are dropped. If it is
> false, the second argument of assert is evaluated and its value together
> with all immediate results of the first expression, together with
> references to corresponding subexpressions (as strings, ranges or AST
> nodes) are passed to the special handler. That handler can be
> implemented in a third-party library, because formatting and outputting
> a report is a complex task. The default handler can just raise an
> AttributeError.
Is that supposed to be AssertionError?
> It is important to note that expressions with side effects are
> affected by this feature. This is because in order to display this
> information, we must store references to the instances and not just
> the values.
Don't you mean that expressions with side-effects are *not* affected? Or
have I misunderstood?
You don't have to re-evaluate the expression if the assertion fails, so
any side-effects only occur once.
greeting = "Hello world"
assert print(greeting)
Hello world
Assertion failed:
print(greeting)
| |
| 'Hello world'
|
None
Rather than the situation where the expression has to be evaluated
twice (as older versions of Pytest used to do):
greeting = "Hello world"
assert print(greeting)
Hello world
Assertion failed:
print(greeting)
Hello world
| |
| 'Hello world'
|
None
--
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/TMD6RNVZPFTMYP3F2D3ECUAAI5EURTMD/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP Draft: Power Assertion
On Fri, Sep 24, 2021 at 07:45:41PM -0700, Christopher Barker wrote: > 1) this should be part of a test framework, not something built in to > exceptions. Pytest has shown that it can be done without any language > changes. Pytest needs to take heroic measures in an import hook to re-write your assertions to do what it does: http://pybites.blogspot.com/2011/07/behind-scenes-of-pytests-new-assertion.html Maybe a language change, with a public API for assertions, would make it much easier and less hacky for test frameworks to do what Pytest does. It wouldn't be the first time that Python added features to the language to make it easier for third-party libraries. Also, Pytest only supports CPython and PyPy. https://pytest.org/en/6.2.x/getting-started.html I know that Jython and IronPython aren't getting all the love that maybe they should be getting (migration to 3.x seems to be pretty slow), but it would be nice if better assertions was a language feature rather than requiring heroic import hook and AST hacks to work. > 2) there are a lot of things in Pytest that are very useful. Will they be less useful if Python gives better diagnostics on assertion failures? > And a lot of problems with unittest. Such as? The only really unpleasant part of unittest is having to remember the spellings of all the various `self.assertSpam...` methods. Otherwise, it's a pretty nice framework. If assertions were better, perhaps we could fix that. On the other hand, I personally regularly use plain old assert in my unit tests, to distinguish between internal checks of the test logic, versus actual unit tests. Tests are code too, and can be buggy. I think it is important that test code includes assertions that refer to the test code itself. -- Steve ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/A7SKAQBCJU3TW4VI5LESKUJPPT7NCSG2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP Draft: Power Assertion
On Fri, Sep 24, 2021 at 19:49 Christopher Barker wrote: > Alternatively, take the approach taken with distutils and setuptools— > officially accept that a full featured test framework will be left to third > parties. > I think this is by far the best option. Pytest can evolve much faster than the stdlib. —Guido > -- --Guido (mobile) ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5X7ESOCGBP25YNEF3F3XWVDQHOV37N6R/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP Draft: Power Assertion
My first thought when reading this (and the recent discussion on Python ideas) was that pytest already does this. Then I see in the PEP: "What pytest does is awesome“ So what? Well, two points: 1) this should be part of a test framework, not something built in to exceptions. Pytest has shown that it can be done without any language changes. 2) there are a lot of things in Pytest that are very useful. And a lot of problems with unittest. So maybe what we should do now, rather than add one feature, is propose a new test framework/test runner for the stdlib, inspired by pytest. Alternatively, take the approach taken with distutils and setuptools— officially accept that a full featured test framework will be left to third parties. NOTE: if the proposal does require actual language changes beyond the current introspection options, that should be made clear. -CHB On Fri, Sep 24, 2021 at 4:12 AM Paul Moore wrote: > On Fri, 24 Sept 2021 at 12:05, Noam Tenne wrote: > > > Caveats > > --- > > > > It is important to note that expressions with side effects are affected > by this feature. This is because in order to display this information, we > must store references to the instances and not just the values. > > One immediate thought. You should give examples of the sort of > expressions that are affected, and precisely what the effect is. It's > impossible to judge the importance of this point without details. > > Paul > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/JJ2CT4FYFHJOOSABQOD6KRKCQ22FJ2BI/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/M4U5VQ4NUQSCOHHGWRLIO6KUMYI7HVIW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP Draft: Power Assertion
On Fri, 24 Sept 2021 at 12:05, Noam Tenne wrote: > Caveats > --- > > It is important to note that expressions with side effects are affected by > this feature. This is because in order to display this information, we must > store references to the instances and not just the values. One immediate thought. You should give examples of the sort of expressions that are affected, and precisely what the effect is. It's impossible to judge the importance of this point without details. Paul ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/JJ2CT4FYFHJOOSABQOD6KRKCQ22FJ2BI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] PEP Draft: Power Assertion
Hi All,
Following the discussions on "Power Assertions: Is it PEP-able?", I've drafted
this PEP.
Your comments are most welcome.
PEP:
Title: Power Assertion
Author: Noam Tenne
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 24-Sep-2021
Abstract
This PEP introduces a language enhancement named "Power Assertion".
The Power Assertion is inspired by a similar feature found in the
`Groovy language`_ and is an extension to the core lib's ``assert``
keyword.
When an assertion expression evaluates to ``False``, the output shows
not only the failure, but also a breakdown of the evaluated expression
from the inner part to the outer part.
.. _Groovy language:
http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#_power_assertions
Motivation
=
Every test boils down to the binary statement "Is this true or false?",
whether you use the built-in assert keyword or a more advanced
assertion method provided by a testing framework.
When an assertion fails, the output is binary too —
"Expected x, but got y".
There are helpful libraries like Hamcrest which give you a more
verbose breakdown of the difference and answer the question
"What exactly is the difference between x and y?".
This is extremely helpful, but it still focuses on the difference
between the values.
Keep in mind that a given state is normally an outcome of a series of
states, that is, one outcome is a result of multiple conditions and causes.
This is where Power Assertion comes in. It allows us to better
understand what led to the failure.
The Community Wants This
As mentioned in the abstract, this feature was borrowed from Groovy.
It is a very popular feature within the Groovy community, also used by
projects such as `Spock`_.
On top of that, it is very much needed in the Python community as well:
* `Power Assertion was explicitly requested`_ as a feature in the
`Nimoy`_ testing framework
* There's a `similar feature in pytest`_
And when `discussed in the python-ideas`_ mailing list, the responses
were overall positive:
* "This is cool." - Guido van Rossum
* "I was actually thinking exactly the opposite: this would more
useful in production than in testing."
- 2QdxY4RzWzUUiLuE＠potatochowder.com
* "What pytest does is awesome. I though about implementing it in the
standard compiler since seen it the first time." - Serhiy Storchaka
.. _Spock: https://spockframework.org/
.. _Power Assertion was explicitly requested:
https://stackoverflow.com/a/58536986/198825
.. _similar feature in pytest:
https://docs.pytest.org/en/latest/how-to/assert.html
.. _discussed in the python-ideas:
https://mail.python.org/archives/list/python-ideas@python.org/thread/T26DR4BMPG5EOB3A2ELVEWQPYRENRXHM/
Rational
Code Example
::
class SomeClass:
def __init__(self):
self.val = {'d': 'e'}
def __str__(self):
return str(self.val)
sc = SomeClass()
assert sc.val['d'] == 'f'
This routine will result in the output:
::
Assertion failed:
sc.val['d'] == f
| ||
| eFalse
|
{'d': 'e'}
Display
---
In the output above we can see the value of every part of the
expression from left to right, mapped to their expression fragment
with the pipe (``|``).
The number of rows that are printed depend on the value of each
fragment of the expression.
If the value of a fragment is longer than the actual fragment
(``{'d': 'e'}`` is longer than ``sc``), then the next value (``e``)
will be printed on a new line which will appear above.
Values are appended to the same line until it overflows in length to
horizontal position of the next fragment.
This way of presentation is clearer and more human friendly than the
output offered by pytest's solution.
The information that’s displayed is dictated by the type.
If the type is a constant value, it will be displayed as is.
If the type implements `__str__`, then the return value of that will
be displayed.
Mechanics
-
Reference Implementation
The reference implementation uses AST manipulation because this is
the only way that this level of involvement can be achieved by a
third party library.
It iterates over every subexpression in the assert statement.
Subexpressions are parts of the expression separated by a lookup
(``map[key]``), an attribute reference (``key.value``) or a binary
comparison operator (``==``).
It then builds an AST in the structure of a tree to maintain
the order of the operations in the original code, and tracks the
original code of the subexpression together with the AST code of the
subexpression and the original indices.
It then rewrites the AST of the original expression to call a
specialised assertion function, which accepts the tree as a parameter.
At runtime the expression is executed. If it fails, a rendering
function is called to render the assertion message
