[Python-ideas] Re: Python should take a lesson from APL: Walrus operator not needed

[Marko Ristin-Kaufmann]

Hi,

>  I mean, as shown in this example and a previous one I posted a screenshot
> of, I think it's cute and geeky to use a few math symbols in the same way
> in my editor.  I've been doing that for a few years, and it never got
> beyond "slightly cute."
>

I  would second this. I find it actually less readable if the font does not
provide nice arrows. It reminds me of ScaLa and the "=>" symbol. The right
implication arrow was barely readable in most common Ubuntu fonts.

Cheers,
Marko

>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RUC6NH26QBEVGAK575A4SABU7A7KNIRF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Need a core developer sponsor for a PEP

[Marko Ristin-Kaufmann]

Hi Philippe,
Thanks for preparing the PEP! I also think it will improve the readability
a lot in code that heavily uses type annotations.

I'd suggest you to split this PEP in two: one for Union types and another
one for Optional. It will facilitate the discussions and reaching consensus
since these are actually two separate issues. For example, I endorse the
proposal for "|" operator, while I have some reservations about "~" (e.g.,
~~x is expected to be x?).

I'm not a core developer, though, so maybe they prefer joint PEPs rather
than seperate ones.

Cheers,
Marko

Le mar. 17 sept. 2019 à 08:56, Philippe Prados 
a écrit :

> Hello,
>
> I would like to publish a new PEP (see here
> ).
> I discussed this idea here
> 
> .
> The PEP1  explains that I must
> have a sponsor.
>
> Who can help me ?
>
> Thanks
>
> Philippe Prados
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/WWSOVKGMBCDQBUNJDJYKYKE7FVWHSDU5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/E7XBEGNRQCZTNB44WE4WUUXYZCQAA7YQ/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Discussion: Duck typing with “concepts”

[Marko Ristin-Kaufmann]

Hi James,
As Ivan has mentioned, Protocols already allow for statical type checks:

https://mypy.readthedocs.io/en/latest/protocols.html

We didn't need protocols that often at Parquery, maybe half a dozen of
times?

While we didn't use them in Python, we had to use them intensively in Go
where it is a bit of a nightmare. It gives you freedom for cases when your
input arguments are fairly general (e.g., in a public library), but it made
refactoring our _production_ code (i.e. specific in contrast to general)
much harder since you couldn't look up easily which type implements which
"interface" (as protocols are called in Go).

Cheers Marko
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] [Brainstorm] Testing with Documented ABCs

[Marko Ristin-Kaufmann]

Hi Abe,

I agree. That's why I prefaced this topic with [Brainstorm]. I want to
> explore the solution space to this problem and discuss some of the pros and
> cons of different ideas,  *not* proceed straight to action.


You are right. Please apologize, I was so primed by the discussions we had
in October 2019 that I didn't pay enough attention to "branstorm" in the
subject.

Fuzz testing and stateful testing like that provided by hypothesis might
> work together with contracts in an interesting way.
>

You might want to look at the literature on automatic test generation. A
possible entry point could be:
https://www.research-collection.ethz.ch/handle/20.500.11850/69581

If I had time available, I would start with a tool that analyses a given
module and automatically generates code for the Hypothesis test cases. The
tool needs to select functions which accept primitive data types and for
each one of them translates their contracts into Hypothesis code. If
contracts are not trivially translatable to Hypothesis, the function is
ignored. For readability and speed of development (of the code under test,
not of the tool), I would prefer this tool *not *to be dynamic so that the
developer herself needs to re-run it if the function signatures changed.

The ingredients for such a tool are all there with icontract (similar to
sphinx-icontract, you import the module and analyze its functions; you can
copy/past parts of sphinx-icontract implementation for parsing and listing
the AST of the contracts). (If you'd like to continue discussing this
topic, let's create an issue on icontract github page or switch to private
correspondence in order not to spam this mail list).

There seems like a lot of opportunity for the re-use of contracts, so maybe
> we should consider a mechanism to facilitate that.
>

This was the case for the requests library. @James Lu 
was looking into it -- a lot of functions had very similar contracts.
However, in our code base at the company (including the open-sourced
libraries), there was not a single case where we thought that contracts
re-use would be beneficial. Either it would have hurt the readability and
introduce unnecessary couplings (when the contracts were trivial) or it
made sense to encapsulate more complex contracts in a separate function.


>> *Multiple predicates per decorator. *
>>
> I suppose it may be difficult to implement a clean, *backwards-compatible*
> solution, but yes; going through the arguments in a sequence would be my
> naive solution. Each entry has an optional description, a callable, and an
> optional tag or level to enable toggling (I would follow a simple model
> such as logging levels) *in that order*.
>

I found that to be too error-prone in a larger code base, but that is my
very subjective opinion. Maybe you could make an example?

but without new syntax; each step between icontracts and an Eiffel-esque
> platonic ideal would require significant hackery with diminishing returns
> on investment.
>

I agree. There are also issues with core python interpreter which I expect
to remain open for a long time (see the issues related to retrieving code
text of lambda functions and decorators and tweaking dynamically the
behavior of help(.) for functions).

Cheers,
Marko

>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] [Brainstorm] Testing with Documented ABCs

[Marko Ristin-Kaufmann]

pre(x >= 0,
> y >= 0,
> width >= 0,
> height >= 0,
> x + width <= width_of(img),
> y + height <= height_of(img))
>
> # this would probably be declared at the class level
> inv(*lambda* self: (self.x, self.y) in self,
> *lambda* self: (self.x+self.width-1, self.y+self.height-1) in
> self,
> *lambda* self: (self.x+self.width, self.y+self.height) not in
> self)
>
> self.img = img[y : y+height, x : x+width].copy()
> self.x = x
> self.y = y
> self.width = width
> self.height = height
>
> That might be super tricky to implement, but it saves you some lambda
> noise. Also, I saw a forked thread in which you were considering some sort
> of transpiler  with similar syntax to the above example. That also works.
> Another thing to consider is that the role of descriptors
> <https://www.smallsurething.com/python-descriptors-made-simple/> overlaps
> some with the role of invariants. I don't know what to do with that
> knowledge, but it seems like it might be useful.
>
> Anyway, I hope those half-baked thoughts have *some* value...
>
> On Wed, Nov 28, 2018 at 1:12 AM Marko Ristin-Kaufmann <
> marko.ris...@gmail.com> wrote:
>
>> Hi Abe,
>>
>> I've been pulling a lot of ideas from the recent discussion on design by
>>> contract (DBC), the elegance and drawbacks
>>> <https://bemusement.org/doctests-arent-code> of doctests
>>> <https://docs.python.org/3/library/doctest.html>, and the amazing talk
>>> <https://www.youtube.com/watch?v=MYucYon2-lk> given by Hillel Wayne at
>>> this year's PyCon entitled "Beyond Unit Tests: Taking your Tests to the
>>> Next Level".
>>>
>>
>> Have you looked at the recent discussions regarding design-by-contract on
>> this list (
>> https://groups.google.com/forum/m/#!topic/python-ideas/JtMgpSyODTU
>> and the following forked threads)?
>>
>> You might want to have a look at static checking techniques such as
>> abstract interpretation. I hope to be able to work on such a tool for
>> Python in some two years from now. We can stay in touch if you are
>> interested.
>>
>> Re decorators: to my own surprise, using decorators in a larger code base
>> is completely practical including the  readability and maintenance of the
>> code. It's neither that ugly nor problematic as it might seem at first look.
>>
>> We use our https://github.com/Parquery/icontract at the company. Most of
>> the design choices come from practical issues we faced -- so you might want
>> to read the doc even if you don't plant to use the library.
>>
>> Some of the aspects we still haven't figured out are: how to approach
>> multi-threading (locking around the whole function with an additional
>> decorator?) and granularity of contract switches (right now we use
>> always/optimized, production/non-optimized and teating/slow, but it seems
>> that a larger system requires finer categories).
>>
>> Cheers Marko
>>
>>
>>
>>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] [Brainstorm] Testing with Documented ABCs

[Marko Ristin-Kaufmann]

Hi,

Property based testing is not about just generating random values till the
> heath death of the universe, but generating sensible values in a
> configurable way to cover all equivalence classes we can think of. if my
> function takes two floating point numbers as arguments, hypothesis
> "strategies" won't try all possible combinations of all possible floating
> point values, but instead all possible combination of interesting values
> (NaN, Infinity, too big, too small, positive, negative, zero, None, decimal
> fractions, etc..), something that an experienced programmer probably would
> end up doing by himself with a lot of test cases, but that can be better
> done with less effort by the automation provided by the hypothesis package.
>

Exactly. A tool can go a step further and, based on the assertions and
contracts, generate the tests automatically or prove that certain
properties of the program always hold. I would encourage people interested
in automatic testing to have a look at the scientific literature on the
topic (formal static analysis). Abstract interpretation has been already
mentioned: https://en.wikipedia.org/wiki/Abstract_interpretation. For some
bleeding edge, have a look what they do at this lab with the machine
learning: https://eth-sri.github.io/publications/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] [Brainstorm] Testing with Documented ABCs

[Marko Ristin-Kaufmann]

Hi Abe,

I've been pulling a lot of ideas from the recent discussion on design by
> contract (DBC), the elegance and drawbacks
>  of doctests
> , and the amazing talk
>  given by Hillel Wayne at
> this year's PyCon entitled "Beyond Unit Tests: Taking your Tests to the
> Next Level".
>

Have you looked at the recent discussions regarding design-by-contract on
this list (
https://groups.google.com/forum/m/#!topic/python-ideas/JtMgpSyODTU
and the following forked threads)?

You might want to have a look at static checking techniques such as
abstract interpretation. I hope to be able to work on such a tool for
Python in some two years from now. We can stay in touch if you are
interested.

Re decorators: to my own surprise, using decorators in a larger code base
is completely practical including the  readability and maintenance of the
code. It's neither that ugly nor problematic as it might seem at first look.

We use our https://github.com/Parquery/icontract at the company. Most of
the design choices come from practical issues we faced -- so you might want
to read the doc even if you don't plant to use the library.

Some of the aspects we still haven't figured out are: how to approach
multi-threading (locking around the whole function with an additional
decorator?) and granularity of contract switches (right now we use
always/optimized, production/non-optimized and teating/slow, but it seems
that a larger system requires finer categories).

Cheers Marko
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Contracts in python -- a report & next steps

[Marko Ristin-Kaufmann]

Hi,

Stephen J. Turnbull wrote:

> You can't weaken the
> contracts for iparent.recontract as far as I can see in a decorator-
> based contract library


Of course you can weaken the contracts for iparent.recontract with a
decorator-based contract library. From my first message:

> The features include:
> ...
>
* inheritance of the contracts (with strengthening/weakening)
> ...
>

Please also have a look at the readme on
https://github.com/Parquery/icontract/, Section "Inheritance".

Chris Barker wrote:

> However, I'm not sure it's anywhere near time to actually do that --
> before we get there, there needs to be a pretty good community of folks
> using icontract (or maybe something else?) and ideally some interest from a
> core developer or two.


I absolutely agree. In my first message I wrote:

> Before we organize a collective to write a proposal to standardize the
> library, I would suggest that a couple of other interested teams adopt
> icontract, apply it to their code bases and report their experiences on
> this mail list. I hope that 2-3 reports would be insightful enough to
> either convince other people that contracts in python are worth
> standardizing (and highlight missing features yet to be implemented) or
> provide solid material to discard the endeavor at the current moment.
>

and in my second message (before yours), I wrote:

> My conclusion is at this point is that the best course of action would be
> that other teams pick up a contract library (now that there is at least
> icontract library with a full feature set for design-by-contract akin to
> Eiffel and other languages), and present us how they use contracts and what
> problems they were facing (*e.g., *were there any problems with the
> toggling granularity? readability? maintainability?). Then we decide how to
> proceed.
>

Stefane Fermigier wrote:

> 1) Just as with with PEP 484, we could distinguish between contract
> specification and checking. To achieve some of the goals that you state, we
> need to standardise the way people would state contracts in their code (and
> provide a small implementation in the stdlib, similar to the typing
> module), but how these contracts are verified (at runtime or statically) or
> leveraged in the documentation are left to third party projects (similar to
> mypy & al).
>

This is a great idea -- though I'm not sure how it could be implemented
other than already specifying the implementation (i.e. how the contracts
are verified). If you have some time, could you elaborate a bit how this
might work? I'm suffering from tunnel vision at this point after
implementing the library.


> 2) Building momentum is paramount. First, since there are many contracts
> libraries out there, some dating more than 15 years (ex: PyDBC), we'd need
> to build consensus between the people who wrote and use these libraries.
> And of course, get feedback from people who use this approach on
> significant projects. I'm willing to try your library on one of my projects.
>

Thanks, I'm looking forward to hearing your experience!

I contacted the authors of dpcontracts, but there is still no clear plan of
action how to merge icontract and dpcontracts (see
https://github.com/deadpixi/contracts/issues/15). Other libraries seem
either unmaintained or not applicable except for single-argument checks and
involve custom syntax (like https://pypi.org/project/PyContracts/; I only
played a bit with PyContracts, so it would be great if anybody could tell
their experience with it in a larger code base).

We found that all the present libraries were far from usable for a larger
code base (so we rolled out our own). To the best of our knowledge, none
could deal with inheritance properly (weakening/strengthening) and you
always had to write out the condition along the lambda function (unless the
code was parsed from the documentation which increases the import time
significantly even if you want to disable some of the contracts). As others
already mentioned in the previous discussion, part of the reason for not
adopting DbC was the lack of tools. I'm not saying that icontract should be
that standard tool -- not at all! We need to first find out what such
standard tool would need to fulfill*. *Assuming that we don't make language
changes, what would we like the standard contract library to look like?
What features should it have? What is too much of a niche to be left out?


> 3) Having contracts for most of the stdlib would be a worthy goal (for
> contracts users) but, as noted by some, would come with some speed issues.
> And of course, would represent a multi-year effort. Hopefully your project
> (i.e. standardising a library, but not "contracting" the stdlib) would
> still be valuable if this never happen.
>

I agree.

Nathaniel Smith wrote:

> In your position, I wouldn't be talking to the core devs; I'd be
> writing blog posts to proselytize the advantages of contracts, working
> with popular projects that are interested in 

Re: [Python-ideas] Contracts in python -- a report & next steps

[Marko Ristin-Kaufmann]

Hi Chris,

If not, all your proposed benefits can be achieved at the level of a
> single project, by just saying "we're going to use THIS contracts
> library", unless I'm misunderstanding something here.
>


I think we are having a big disconnect in the discussion. Please apologize
for my vagueness and point me where you would like me to elaborate more.
Let me try to have another iteration where I'll try to paraphrase myself
and illustrate what I think that you are thinking.

I imagine that you conceive contracts purely as an approach to a testing to
be applied to a single project. I'm not talking about that. I'm talking
about two packages on pypi, both specifying contracts, each developed by a
separate team from different organizations. Let's call these packages
package_a and package_b, respectively (and team A and B, analogously).

Imagine now that the contracts are not standardized. For example, team A
uses dpcontracts whereas team B uses icontract. Enter team C working on the
package_c.

There is a class package_c.SomeClass that needs to inherit both from
package_a.some_module.BaseClass and from
package_b.another_module.AnotherBaseClass. Imagine the mess that developers
in team C have to deal with -- how are contracts supposed to be
strengthened and weakened in package_c.SomeClass? A nightmare with
different contract libraries. Even if they don't have to work with multiple
inheritance, they would need to use different syntaxes for different
concrete classes inheriting from the two packages, respectively. And now
think of contract groups that can be specified and applied to functions:

@require(...)
@require(...)
@ensure(...)
def some_func(...):
...

@enforce_contracts_of(some_func)
@require(...)
def some_other_func(...):
...

some_group = [
require(...),
ensure(...),
ensure(...)
]

@enforce_contracts(some_group)
@ensure(...)
def yet_another_function(...):
...


How are these contract groups supposed to play together between package_a
and package_b when applied in package_c? (A side remark: the name and
implementation of "enforce_contracts_of" and "enforce_contracts" are still
in discussion for icontract and have not been implemented yet.)

Furthermore, what about toggling contracts? Team C needs to know both how
to toggle the contracts of package_a.some_module and also how to toggle the
contracts of package_b.another_module. They need to be aware of and
probably set two different environment variables with different toggling
granularities *etc*. What if the toggling conflicts?

Even if they both used, say, icontract. Package_a might require icontract
1.x whereas package_b requires 2.x. This results in a dependency hell. A
standardized lib would force both packages to use the same version of
contract lib (or not be supported in that given python version at all) --
the compatibility would be dictated by the python version, not by the
version of a non-standard contract lib.

Let's focus now on the team working on mypy. For example, consider the code:

@invariant(lambda self: self.something is not None or self.another is not None)
class SomeClass:
...

# later in code somewhere
def some_func():
s = SomeClass(...)
if s.something is None:
# mypy knows here that s.another is not None.
...

But if there existed multiple non-standard contract libraries, can we
really expect that mypy deals with all of them? I couldn't possibly imagine
that.

If these points are still not clear -- imagine having multiple
non-standard, but widely used libraries and approaches for abstract base
classes or object-oriented programming. A horrifying thought :) I'd
probably even prefer the python world without contracts than with a
widespread use of multiple contract libraries.

If pythonistas are to use contracts more widely without stomping on each
other's feet, there needs to be some form of standardization so that
different packages can inter-operate. There needs to be a convention how to
toggle the contracts at different levels of granularity (individual
contracts, module-level, pypi package-level, only in testing *etc.*).
Environment variables? Setting global variables before importing a module?
Some mechanism I'm not aware of? Are the contracts on or off by default?

There are so many other details that needs to be fleshed out and this needs
to be a collective effort if it is to succeed. I don't have the knowledge
to figure out all the conceptual details alone and need help from more
experienced people.

---

Now a completely different point. Whether we write contracts *for* the
modules of the standard library is another discussion. It is not directly
connected with the standardization of contracts (and also not part of the
current thread, IMO). To write contracts or not to write contracts for a
standard library is a per-module decision and concerns the authors and
users of each module in separation. Some modules might include contracts
that are run only in test mode (*e.g., 

Re: [Python-ideas] Contracts in python -- a report & next steps

[Marko Ristin-Kaufmann]

Hi Chris,

For the sake of those of us who REALLY don't feel like diving back
> into the extensive threads on this subject, can you please summarize
> the benefits of having this in the stdlib rather than as a third-party
> library?
>

Certainly. We need a standard approach to contracts as opposed to
third-party libraries for the following technical reasons:
* There is no dependency hell if two packages use different versions of the
same contract library.
* Two packages need to inherit each other's contracts (*e.g.*, one package
defines a class which inherits a base class from a different package and
hence needs to inherit its contracts as well).
* Third-party libraries for testing and static verification need a standard
approach to contracts in order to be usable. Otherwise, the authors of
these libraries (*e.g. *Hypothesis) need to support multiple contrat
libraries (if they could ever bother to support multiple of them).

There are many more, but these are the reasons that I find critical.


> Also - have you benchmarked the performance cost of adding contracts?
> Particularly: if you're planning to contractify the stdlib, what is
> the impact on startup performance?
>

I made some preliminary benchmarks. The source code is available at:
https://github.com/Parquery/icontract/tree/master/benchmarks/startup

Average over ten runs in milliseconds on my machine (Intel(R) Core(TM)
i7-4700MQ CPU @ 2.40GHz, 8 cores, 4 GB RAM):
Duration to import the module functions_100_with_no_contract : 795.59 ±
10.47

Duration to import the module functions_100_with_1_contract : 919.53 ± 61.22
Duration to import the module functions_100_with_5_contracts : 1075.81 ±
59.87
Duration to import the module functions_100_with_10_contracts : 1290.22 ±
90.04

Duration to import the module functions_100_with_1_disabled_contract :
833.60 ± 32.07
Duration to import the module functions_100_with_5_disabled_contracts :
851.31 ± 66.93
Duration to import the module functions_100_with_10_disabled_contracts :
897.90 ± 143.02

Duration to import the module classes_100_with_no_invariant : 843.61 ± 28.21

Duration to import the module classes_100_with_1_invariant : 3409.71 ± 95.78
Duration to import the module classes_100_with_5_invariants : 4005.93 ±
131.97
Duration to import the module classes_100_with_10_invariants : 4801.82 ±
157.56

Duration to import the module classes_100_with_1_disabled_invariant :
885.88 ± 44.24
Duration to import the module classes_100_with_5_disabled_invariants :
912.53 ± 101.91
Duration to import the module classes_100_with_10_disabled_invariants :
963.77 ± 161.76

Please let me know if these are the benchmarks you had in mind or you would
like to see something else. I have neither optimized the code for speed nor
investigated why the performance of class invariants differs so much from
the functions. The important bit for me was that disabling contracts
basically had no impact on import time.

Cheers,
Marko
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Contracts in python -- a report & next steps

[Marko Ristin-Kaufmann]

P.S. Here is the link to the github repo:
https://github.com/Parquery/icontract

On Wed, 24 Oct 2018 at 09:40, Marko Ristin-Kaufmann 
wrote:

> Hi,
> I would like to give you a short report on the development of icontract
> library following the discussion about the introduction of contracts into
> Python (e.g., see [1, 2, 3, 4]).
>
> *Features*
> The functionality of icontract library is basically on par with Eiffel,
> Cobra and other languages supporting contracts except for loop invariants
> and less readable syntax.
>
> The features include:
> * "require", "ensure" and "invariant" decorators to define the contracts
> * "snapshot" decorator to capture the values prior to the function
> invocation to allow postconditions to verify the state transitions
> * inheritance of the contracts (with strengthening/weakening)
> * tracing of the argument values on contract violation
> * informative violation messages automatically parsed from the code of the
> condition function
> * The user can specify custom errors to be raised on a specific contract
> violation
> * individual toggling of the contracts
> * linter to check the contract arguments
> * sphinx extension to render the contracts automatically in the
> documentation (including a sophisticated matching of logical implications)
>
> We covered all the use cases we could find in our code base (at Parquery
> AG) such as:
> * functions
> * instance methods
> * class and static methods
> * property getters, setters and deleters
> * slot wrappers
>
> *Roadblocks*
> During the development, the following roadblocks were encountered:
>
> * We wanted to include the contracts in the output of help().
> Unfortunately, help() renders the __doc__ of the class and not of the
> instance. For functions, this is the class "function" which you can not
> inherit from. See [5] for more details.
>
> * We need to inspect the source code of the condition and error lambdas to
> generate the violation message and infer the error type in the
> documentation, respectively. inspect.getsource(.) is broken on lambdas
> defined in decorators in Python 3.5.2+ (see [6]). We circumvented this bug
> by using inspect.findsource(.), inspect.getsourcefile(.) and examining the
> local source code of the lambda by searching for other decorators above and
> other decorators and a function or class definition below. The decorator
> code is parsed and then we match the condition and error arguments in the
> AST of the decorator. This is brittle as it prevents us from having partial
> definitions of contract functions or from sharing the contracts among
> functions.
>
> Here is a short code snippet to demonstrate where the current
> implementation fails:
> import icontract
>
> require_x_positive = icontract.require(
> lambda x: x > 0, error=lambda: ValueError("x must be positive"))
>
> @require_x_positive
> def some_func(x: int) -> None:
> pass
>
> However, we haven't faced a situation in the code base where we would do
> something like the above, so I am unsure whether this is a big issue. As
> long as decorators are directly applied to functions and classes,
> everything worked fine on our code base.
>
>
> *Our Experience*
> We have been using icontract in our team for our production code base
> (~100K LOC) as well as for a couple of open source projects (each <10K LOC)
> since August 1st 2018 (~ 3 months).
>
> In contrast to points raised during the discussions in [1, 2, 3, 4], we
> did not have issues with readability and modifying contracts. Thus far, we
> have not encountered problems with variable renames and major code
> refactorings. We use Pycharm, so it remains open whether this also applies
> to development environments such as emacs and vim.
>
> We confirm that contracts help us improve the documentation, keep the
> documentation up-to-date, better reason about the function's input and
> output and catch bugs early and more easily during the unit and integration
> tests.
>
>
> *Status*
> The library interface is currently frozen after the version bump to 2.0.0
> and is not expected to change in the next six months. All the reported bugs
> have been fixed and no bug fixes are pending.
>
> *Next Steps?*
> I personally doubt that we are enough people to form a party to push for a
> change in the language. A standardized library seems to me like a
> realizable compromise given the state of the discussion on this mail list.
>
> Before we organize a collective to write a proposal to standardize the
> library, I would suggest that a couple of other interested teams adopt
> icontract, apply it to their code bases and report t

[Python-ideas] Contracts in python -- a report & next steps

[Marko Ristin-Kaufmann]

Hi,
I would like to give you a short report on the development of icontract
library following the discussion about the introduction of contracts into
Python (e.g., see [1, 2, 3, 4]).

*Features*
The functionality of icontract library is basically on par with Eiffel,
Cobra and other languages supporting contracts except for loop invariants
and less readable syntax.

The features include:
* "require", "ensure" and "invariant" decorators to define the contracts
* "snapshot" decorator to capture the values prior to the function
invocation to allow postconditions to verify the state transitions
* inheritance of the contracts (with strengthening/weakening)
* tracing of the argument values on contract violation
* informative violation messages automatically parsed from the code of the
condition function
* The user can specify custom errors to be raised on a specific contract
violation
* individual toggling of the contracts
* linter to check the contract arguments
* sphinx extension to render the contracts automatically in the
documentation (including a sophisticated matching of logical implications)

We covered all the use cases we could find in our code base (at Parquery
AG) such as:
* functions
* instance methods
* class and static methods
* property getters, setters and deleters
* slot wrappers

*Roadblocks*
During the development, the following roadblocks were encountered:

* We wanted to include the contracts in the output of help().
Unfortunately, help() renders the __doc__ of the class and not of the
instance. For functions, this is the class "function" which you can not
inherit from. See [5] for more details.

* We need to inspect the source code of the condition and error lambdas to
generate the violation message and infer the error type in the
documentation, respectively. inspect.getsource(.) is broken on lambdas
defined in decorators in Python 3.5.2+ (see [6]). We circumvented this bug
by using inspect.findsource(.), inspect.getsourcefile(.) and examining the
local source code of the lambda by searching for other decorators above and
other decorators and a function or class definition below. The decorator
code is parsed and then we match the condition and error arguments in the
AST of the decorator. This is brittle as it prevents us from having partial
definitions of contract functions or from sharing the contracts among
functions.

Here is a short code snippet to demonstrate where the current
implementation fails:
import icontract

require_x_positive = icontract.require(
lambda x: x > 0, error=lambda: ValueError("x must be positive"))

@require_x_positive
def some_func(x: int) -> None:
pass

However, we haven't faced a situation in the code base where we would do
something like the above, so I am unsure whether this is a big issue. As
long as decorators are directly applied to functions and classes,
everything worked fine on our code base.


*Our Experience*
We have been using icontract in our team for our production code base
(~100K LOC) as well as for a couple of open source projects (each <10K LOC)
since August 1st 2018 (~ 3 months).

In contrast to points raised during the discussions in [1, 2, 3, 4], we did
not have issues with readability and modifying contracts. Thus far, we have
not encountered problems with variable renames and major code refactorings.
We use Pycharm, so it remains open whether this also applies to development
environments such as emacs and vim.

We confirm that contracts help us improve the documentation, keep the
documentation up-to-date, better reason about the function's input and
output and catch bugs early and more easily during the unit and integration
tests.


*Status*
The library interface is currently frozen after the version bump to 2.0.0
and is not expected to change in the next six months. All the reported bugs
have been fixed and no bug fixes are pending.

*Next Steps?*
I personally doubt that we are enough people to form a party to push for a
change in the language. A standardized library seems to me like a
realizable compromise given the state of the discussion on this mail list.

Before we organize a collective to write a proposal to standardize the
library, I would suggest that a couple of other interested teams adopt
icontract, apply it to their code bases and report their experiences on
this mail list. I hope that 2-3 reports would be insightful enough to
either convince other people that contracts in python are worth
standardizing (and highlight missing features yet to be implemented) or
provide solid material to discard the endeavor at the current moment.

In the meanwhile, it would be of great help if somebody could vet the
documentation and the code of icontract.

*Authors*
The library was mainly written by me (with some help of my colleague Adam
Radomski). We discussed the features within the dev team at Parquery
(Zurich, Switzerland) as well as in email correspondence with James Lu.

[1] 

Re: [Python-ideas] Multi Statement Lambdas

[Marko Ristin-Kaufmann]

Hi,
What about writing longer map in pyspark? When I worked with Spark
and Scala, it was easy to script a longer chain of transformations in
Scala. Does anybody know how that works in Python without multiline lambdas?

Cheers Marko


Le dim. 21 oct. 2018 à 18:40, Ron Reiter  a écrit :

> Multi-line lambdas may be nice for functional languages, but Python is
> very imperative in nature and prefers clean syntax over conciseness, which
> means it will make it REALLY hard to fit with Python. I personally find
> multi-line lambdas an unreadable abomination.
>
> As for some more concrete reasons on why this should not be a part of
> Python, see here for an example which shows why it would even be hard to
> come up with a plausible syntax for multi-line lambdas:
>
> https://stackoverflow.com/questions/1233448/no-multiline-lambda-in-python-why-not
>
> Guido's words:
>
> "But the complexity of any proposed solution for this puzzle is immense,
> to me: it requires the parser (or more precisely, the lexer) to be able to
> switch back and forth between indent-sensitive and indent-insensitive
> modes, keeping a stack of previous modes and indentation level. Technically
> that can all be solved (there's already a stack of indentation levels that
> could be generalized). But none of that takes away my gut feeling that it
> is all an elaborate Rube Goldberg contraption."
>
> If one would find a syntax which is Pythonic and clean then I am in favor
> of adding it, but I argue it simply does not exist.
>
> - Ron
>
>
> [image: Facebook]  [image: Twitter]
>  [image: LinkedIn]
> 
>
>
> On Sun, Oct 21, 2018 at 7:34 PM Anders Hovmöller 
> wrote:
>
>>
>> Wheres a lambda expression can be passed anonymously to any other
>> function as an argument.
>>
>> *map(lambda x: x**2, array)*
>>
>> vs
>>
>> *def powers2(x)*
>> *   x**2*
>> *map(powers2, array)*
>>
>> Coming up with a name here is not needed, as the operation is expressive
>> enough.
>>
>>
>> Sure, but there is a well known convention for such non-names already:
>> meta syntactical variables. In this case the name could be "foo" and all
>> readers will interpret this as "no name".
>>
>> I admit to have wanted lambdas that are just as powerful as normal
>> functions, but often when I want it the code is nicer when a normal
>> function is used.
>>
>> / Anders
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Chris,

I hope you don't mind me responding though you would like to stop
participating. This message is meant for other readers in case they are
interested.

> Alice tests her package A with some test data D_A. Now assume Betty did
> not write any contracts for her package B. When Alice tests her package,
> she is actually making an integration test. While she controls the inputs
> to B from A, she can only observe the results from B, but not whether they
> are correct by coincidence or B did its job correctly. Let's denote D'_B
> the data that is given to B from her original test data D_A during Alice's
> integration testing.
> >
>
> If you're regularly changing your function contracts, such that you
> need to continually test  in case something in the other package
> changed, then yes, that's exactly what I'm talking about.
>

The user story I put above had nothing to do with change. I was telling how
manually performing integration tests between A and B is tedious for us
(since it involves some form or the other of manual recording of
input/outputs to the module B and adapting unit tests of B) while contracts
are much better (*for us*) since they incur little overhead (write them
once for B, anybody runs them automatically).

I did not want to highlight the *change* in my user story, but the ease of
integration tests with contracts. If it were not for contracts, we would
have never performed them.

Cheers,
Marko



On Mon, 8 Oct 2018 at 16:22, Chris Angelico  wrote:

> On Mon, Oct 8, 2018 at 11:11 PM Steven D'Aprano 
> wrote:
> >
> > On Mon, Oct 08, 2018 at 09:32:23PM +1100, Chris Angelico wrote:
> > > On Mon, Oct 8, 2018 at 9:26 PM Steven D'Aprano 
> wrote:
> > > > > In other words, you change the *public interface* of your functions
> > > > > all the time? How do you not have massive breakage all the time?
> > > >
> > > > I can't comment about Marko's actual use-case, but *in general*
> > > > contracts are aimed at application *internal* interfaces, not so much
> > > > library *public* interfaces.
> > >
> > > Yet we keep having use-cases shown to us involving one person with one
> > > module, and another person with another module, and the interaction
> > > between the two.
> >
> > Do we? I haven't noticed anything that matches that description,
> > although I admit I haven't read every single post in these threads
> > religiously.
>
> Try this:
>
> On Mon, Oct 8, 2018 at 5:11 PM Marko Ristin-Kaufmann
>  wrote:
> > Alice tests her package A with some test data D_A. Now assume Betty did
> not write any contracts for her package B. When Alice tests her package,
> she is actually making an integration test. While she controls the inputs
> to B from A, she can only observe the results from B, but not whether they
> are correct by coincidence or B did its job correctly. Let's denote D'_B
> the data that is given to B from her original test data D_A during Alice's
> integration testing.
> >
>
> If you're regularly changing your function contracts, such that you
> need to continually test  in case something in the other package
> changed, then yes, that's exactly what I'm talking about.
>
> I'm tired of debating this. Have fun. If you love contracts so much,
> marry them. I'm not interested in using them, because nothing in any
> of these threads has shown me any good use-cases that aren't just
> highlighting bad coding practices.
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Crhis,

> In other words, you change the *public interface* of your functions
> > all the time? How do you not have massive breakage all the time?
>
> I can't comment about Marko's actual use-case, but *in general*
> contracts are aimed at application *internal* interfaces, not so much
> library *public* interfaces.
>

Sorry, I might have misunderstood the question -- I was referring to
modules used within the company, not outside. Of course, public libraries
put on pypi don't change their interfaces weekly.

Just to clear the confusion, both Steve and I would claim that the
contracts do count as part of the interface.

For everything internal, we make changes frequently (including the
interface) and more often than not, the docstring is not updated when the
implementation of the function is. Contracts help our team catch breaking
changes more easily. When we change the behavior of the function, we use
"Find usage" in Pycharm, fix manually what we can obviously see that was
affected by the changed implementation, then statically check with mypy
that the changed return type did not affect the callers, and contracts (of
other functions!) catch some of the bugs during testing that we missed when
we changed the implementation. End-to-end test with testing contracts
turned off catch some more bugs on the real data, and then it goes into
production where hopefully we see no errors.

Cheers,
Marko



On Mon, 8 Oct 2018 at 12:32, Chris Angelico  wrote:

> On Mon, Oct 8, 2018 at 9:26 PM Steven D'Aprano 
> wrote:
> > > In other words, you change the *public interface* of your functions
> > > all the time? How do you not have massive breakage all the time?
> >
> > I can't comment about Marko's actual use-case, but *in general*
> > contracts are aimed at application *internal* interfaces, not so much
> > library *public* interfaces.
>
> Yet we keep having use-cases shown to us involving one person with one
> module, and another person with another module, and the interaction
> between the two. Which way is it? Do the contracts change frequently
> or not? Are they public or not? How are we supposed to understand the
> point of contracts if the use-cases being shown all involve bad code
> and/or bad coding practices?
>
> Contracts, apparently, allow people to violate versioning expectations
> and feel good about it.
>
> (Am I really exaggerating all that much here?)
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Chris,

In other words, you change the *public interface* of your functions
> all the time? How do you not have massive breakage all the time?


I suppose that Pycharm helps the most with its refactoring tools. We use
type annotations, contracts, static checks (mypy, pylint, pydocstyle) and
unit, integration and end-to-end tests, so actually unexpected breakages in
production are not that frequent. What does happen often, though, is that
documentation gets stale.

Cheers,
Marko

On Mon, 8 Oct 2018 at 07:29, Chris Angelico  wrote:

> On Mon, Oct 8, 2018 at 4:26 PM Marko Ristin-Kaufmann
>  wrote:
> >> Not true for good docstrings.  We very seldom change the essential
> >> meaning of public functions.
> >
> > In my team, we have a stale docstring once every two weeks or even more
> often. If it weren't for doctests and contracts, I could imagine we would
> have them even more often :)
> >
>
> In other words, you change the *public interface* of your functions
> all the time? How do you not have massive breakage all the time?
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi,
I'd like to share a use pattern for contracts that might have got lost in
the discussion and which I personally grow to like more and more. I'm not
making any claims; this use pattern work for our team and I can't judge how
much of a benefit it would be to others.

Imagine there are two developers, Alice working on a package A, and Betty
working on a package B. Package A depends on package B.

Betty tested her package B with some test data D_B.

Alice tests her package A with some test data D_A. Now assume Betty did not
write any contracts for her package B. When Alice tests her package, she is
actually making an integration test. While she controls the inputs to B
from A, she can only observe the results from B, but not whether they are
correct by coincidence or B did its job correctly. Let's denote D'_B the
data that is given to B from her original test data D_A during Alice's
integration testing.

How can she test that package B gives the correct results on D'_B ? She
needs to manually record the data somehow (by dynamically mocking package B
and intercepting what gets passed from A to B?). She might fetch the tests
from the package B, copy/paste the test cases and append D'_B. Or she could
make a pull request and provide the extra test data directly to package B.
She needs to understand how Betty's unit tests work and see how D'_B fits
in there and what needs to be mocked.

All in all, not a trivial task if Alice is not familiar with the package B
and even less so if Alice and Betty don't work in the same organization.
Most of the time, Alice would not bother to test the dependencies on her
testing data D_A. She would assume that her dependencies work, and just
tests what comes out of them. If the results make sense, she would call it
a tick on her to-do list and move on with the next task.

Let's assume now that Betty wrote some contracts in her code. When Alice
runs the integration test of her package A, the contracts of B are
automatically verified on D'_B. While the contracts might not cover all the
cases that were covered in Betty's unit tests, they still cover some of
them. Alice can be a bit more confident that at least *something* was
checked on D'_B. Without the contracts, she would have checked *nothing* on
D'_B in most of her everyday programming.

You can consider writing contracts as a matter of economy in this story.
Betty might not need contracts for maintaining her package B -- she can
read her code, she can extend her test cases. However, you can see
contracts as a service to the package users, Alice in this case. Betty
helps Alice have some integration tests free-of-charge (free for Alice;
Betty of course pays the overhead of writing and maintaining the
contracts). Alice does not need to understand how B can be tested nor needs
to manually record data that needs to be passed to B. She merely runs her
test code and the checker library will do the testing of B on D'_B
automatically.

The utility of this service tends to grow exponentially in cases where
dependency trees grow exponentially as well. Imagine if we had Carol with
the package C, with the dependencies A -> B -> C. When Carol writes
contracts, she does a service not only to her direct users (Betty) but also
to the users of B (Alice). I don't see how Alice could practically cover
the case with dependencies A -> B -> C and test C with D'_C (*i.e. *test C
with the data coming from D_A) without the contracts unless she really
takes her time and gets familiar with dependencies of all here immediate
dependencies.

We found this pattern helpful in the team, especially during refactorings
where contracts provide an additional security net. We don't have time to
record and add tests of B for D'_B, and even less so of C for D'_C. The
contracts work thus as a good compromise for us (marginal overhead, but
better documentation and "free" integration tests rather than none).

Cheers,
Marko





On Sun, 30 Sep 2018 at 08:17, Marko Ristin-Kaufmann 
wrote:

> Hi,
>
> I compiled a couple of issues on github to provide a more structured
> ground for discussions on icontract features:
> https://github.com/Parquery/icontract/issues (@David Maertz: I also
> included the issue with automatically generated __doc__ in case you are
> still interested in it).
>
> Cheers,
> Marko
>
> On Sat, 29 Sep 2018 at 17:27, Stephen J. Turnbull <
> turnbull.stephen...@u.tsukuba.ac.jp> wrote:
>
>> Steven D'Aprano writes:
>>
>>  > put (x: ELEMENT; key: STRING) is
>>  >  -- Insert x so that it will be retrievable through key.
>>  >  require
>>  >  count <= capacity
>>  >  not key.empty
>>  >  do
>>  >  ... Some insertion algorithm ...
>>  >  ensure
>>  >  has (x)
>>  >  item (key) = x
>>  >

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Terry,

I would encourage you to read all the messages on the thread. The points
you raised were actually already discussed:
* Using formal contracts in the code does not imply that you must specify
*all* the contracts; you specify what you consider meaningful.
* Distinction of result/process (the example that was already discussed was
related to GUIs and pathlib standard library)

> b)//If you write contracts in text, they will become stale over time
>
> Not true for good docstrings.  We very seldom change the essential
> meaning of public functions.


In my team, we have a stale docstring once every two weeks or even more
often. If it weren't for doctests and contracts, I could imagine we would
have them even more often :)

I suppose good docstrings require many reviewers and slow refactoring
processes. Our public interfaces are changing daily and there is one
reviewer per pull request. If github/bitbucket allowed for better
highlighting of docstrings in code reviews (*e.g., *if it highlighted the
docstring of every function that changed), the miss rate would be probably
lower. I'm always happy to hear other experiences how people dealt with
requirements changing at a fast pace and how they dealt with code rot in a
team of limited size and time resources.

I agree with you that a good docstring is not stale by definition. I have
just never experienced a team that had good docstrings in my career.

Cheers,
Marko

On Sun, 7 Oct 2018 at 22:25, Terry Reedy  wrote:

> On 9/24/2018 3:46 AM, Marko Ristin-Kaufmann wrote:
>
> I am responding to your request "Please do point me to what is not
> obvious to you".  I think some of your claims are not only not obvious,
> but are wrong.  I have read some (probably less than half) of the
> responses and avoid saying what I know others have covered adequately.
>
> A mathematical function is defined or specified by a input domain,
> output range, and a mapping from inputs to outputs.  The mapping can be
> defined either by an explicit listing of input-output pairs or by a rule
> specifying either a) the process, what is done to inputs to produce
> outputs or, b) the result, how the output relates to the input.
>
>  >
>
> https://www.win.tue.nl/~wstomv/edu/2ip30/references/design-by-contract/index.html
>
> defines contracts as "precise (legally unambiguous) specifications" (5.2
> Business Contracting/Sub-contracting Metaphor)  It is not obvious to me
> that the metaphor of contracts adds anything worthwhile to the idea of
> 'function'.
>
> 1. Only a small sliver of human interactions are governed by formal
> legal contracts read, understood, and agreed to by both (all) parties.
>
> 2. The idealized definition is naive in practice.  Most legal contracts,
> unlike the example in the link article, are written in language that
> most people cannot read.  Many contracts are imprecise and legally
> ambiguous, which is why we have contract dispute courts.  And even then,
> the expense means that most people who feel violated in a transaction do
> not use courts.
>
> Post-conditions specify a function by result.  I claim that this is not
> always sensible.  I said above that functions may be specified by
> process rather than result.  Ironically, the contract metaphor
> reinforces my claim.  Many contracts, such as in teaching and medicine,
> only specify process and explicitly disclaim any particular result of
> concern to the client.
>
> > b)//If you write contracts in text, they will become stale over time
>
> Not true for good docstrings.  We very seldom change the essential
> meaning of public functions.
>
> How has "Return the sine of x (measured in radians).", for math.sin,
> become stale?  Why would it ever?  What formal executable post condition
> would help someone who does not understand 'sine', or 'sine of x'?  The
> function is only useful for someone who either understands 'sine' or is
> copying a formula written elsewhere without needing to understand it.
>
> Or consider "Stable sort *IN PLACE*." for list.sort.  A formal
> executable post-condition for this *can* be written.  But for someone
> who understands the words 'stable', 'sort', and 'in place', this compact
> English post condition is much more readable than any formal version
> could be.  More over, that addition of 'stable', when we were ready to
> do so, did not make "Sort *IN PLACE*" stale in any sense of being wrong.
>
> I said above that functions definitions and contracts can specify the
> process rather than the result.  For functions, the code is the process,
> and the code may then be the specification and the contract.  For instance,
>
> def append_first(seq):
>  "Append seq[0] to seq."
>
> So seq must have a first elem

Re: [Python-ideas] Dynamic getting of __doc__ of a function

[Marko Ristin-Kaufmann]

Hi Steve,

> > * built-in class function ignoring the property getter (some_func.__doc__
> > set to a property returns the property instead of invoking the getter)
>
> Not just functions, it is all instances. Properties are only invoked if
> they are on the class object, not the instance.
>

I see. The concrete function is simply handled like an instance of the
class "function" and not in any special way.

Is that something I could fix? Or is it still not clear how this should be
fixed? I am totally unaware of the bigger picture and wider repercussions
here.

Cheers Marko

>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Dynamic getting of __doc__ of a function

[Marko Ristin-Kaufmann]

Hi,
@Jonathan: thanks! I'll have a look at your links.

So there are actually two issues if I understand correctly:
* help() ignoring the property getter when invoked on an instance
* built-in class function ignoring the property getter (some_func.__doc__
set to a property returns the property instead of invoking the getter)

Is the second issue also problematic or the built-ins are meant to ignore
properties and there should be no fix? I think I'm misunderstanding
something here.

Cheers,
Marko





Le dim. 7 oct. 2018 à 17:37, Jonathan Fine  a écrit :

> Hi Marko
>
> You wrote:
>
> > I just couldn't figure out how to make the __doc__ attribute of a
> function a getter. Is this a bug or am I making a mistake? If it's my
> mistake, is there any other way how to dynamically __doc__ a function or am
> I forced to set __doc__ of a function at the decoration time?
>
> Thank you for your clear examples. I'm not sure, but I may have found
> the problem.
>
> As I recall, setting obj.attr to a property won't work. Instead, you
> have to set type(obj).attr to the property. And now you come up
> against a problem. You can't set __doc__ on the type 'function', nor
> can you subclass the type 'function'.
>
> >>> def fn(): pass
> >>> type(fn)
> 
>
> >>> type(fn).__doc__
> 'function(code, globals[, name[, argdefs[, closure]]]) [snip]'
>
> >>> type(fn).__doc__ = None
> TypeError: can't set attributes of built-in/extension type 'function'
>
> >>> class FN(type(fn)): pass
> TypeError: type 'function' is not an acceptable base type
>
> As I (more vaguely) recall, something was done recently to add a get
> attribute property that works directly on objects, rather than on
> instances of a class. But I've not been able to find it. Perhaps I
> misunderstood PEP 562.
>
> Some perhaps relevant URLs:
> https://www.python.org/dev/peps/pep-0224/ Attribute Docstrings
> https://www.python.org/dev/peps/pep-0549/ Instance Descriptors
> https://www.python.org/dev/peps/pep-0562/ Module __getattr__ and __dir__
> https://stackoverflow.com/questions/2447353/getattr-on-a-module
>
> --
> Jonathan
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Dynamic getting of __doc__ of a function

[Marko Ristin-Kaufmann]

Hi Crhis,

Have you tried just generating the docstrings at decoration time and
> applying them? By the sound of it, they won't change after that, and
> the only reason to run it later is performance... but have you
> actually measured a performance hit resulting from that?


I did. On my laptop (Intel i7-4700MQ), it takes about 4-12 milliseconds to
parse and reformat a single condition lambda function of a contract. This
quickly adds up to a couple of seconds if you have thousands of condition
functions to parse. This might not be a problem for a limited code base,
but I imagine that it could get inefficient very quickly as soon as the
contract usage would spread to the dependencies as well. Not dead-slow
inefficient, but inefficient enough to consider whether the automatic doc
generation is worth it.

Cheers,
Marko


On Sun, 7 Oct 2018 at 16:46, Chris Angelico  wrote:

> On Mon, Oct 8, 2018 at 1:41 AM Marko Ristin-Kaufmann
>  wrote:
> > (If you wonder about the use case: I'd like to dynamically generate the
> docstrings when functions are decorated with contracts from icontract
> library. Condition functions need to be parsed and re-formatted, so this is
> something that should be done on-demand, when the user either wants to see
> the help() or when the sphinx documentation is automatically generated. The
> docs should not inflict computational overhead during the decoration since
> normal course of program operation does not need pretty-printed contracts.)
> >
>
> Have you tried just generating the docstrings at decoration time and
> applying them? By the sound of it, they won't change after that, and
> the only reason to run it later is performance... but have you
> actually measured a performance hit resulting from that?
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Dynamic getting of __doc__ of a function

[Marko Ristin-Kaufmann]

Hi Steve,

Here are a couple of code snippets. I hope it's not too verbose :)

Setting property on the __doc__ works on the instance, but not on the class:

class A:
@property
def __doc__(self):
return "Works only on the instance"

a = A()
print('A.__doc__ is {}'.format(A.__doc__))
# A.__doc__ is 

print('a.__doc__ is {}'.format(a.__doc__))
# a.__doc__ is Works only on the instance

help(A)
# Help on class A in module __main__:
#
# class A(builtins.object)
#  |  Data descriptors defined here:
#  |
#  |  __dict__
#  |  dictionary for instance variables (if defined)
#  |
#  |  __weakref__
#  |  list of weak references to the object (if defined)


However, you can fix this with a separate class that follows the property
protocol:

some_var = "oi"

class DocProperty:
def __get__(self, instance, owner):
return "Dynamic class doc: {}".format(some_var)

class A:
"""Static doc"""
__doc__ = DocProperty()


print(A.__doc__)
# Dynamic class doc: oi

some_var = "noi"

print(A.__doc__)
# Dynamic class doc: noi

help(A)
# Help on class A in module __main__:
#
# class A(builtins.object)
#  |  Dynamic class doc: noi
#  |
#  |  Data descriptors defined here:
#  |
#  |  __dict__
#  |  dictionary for instance variables (if defined)
#  |
#  |  __weakref__
#  |  list of weak references to the object (if defined)


This worked well when I decorate a *class.* Consider now the case where we
have to decorate a *function*.

I could decorate with a callable class, but __doc__ now does not work and
help() becomes misleading (the function signature is lost and the help says
"Help on Wrapper"):

import functools


class DocProperty:
def __init__(self, func):
self.func = func

def __get__(self, instance, owner):
return "Func doc is: {!r} and now my part...".format(self.func.__doc__)


def decorate(func):
class Wrapper:
__doc__ = DocProperty(func)

def __call__(self, *args, **kwargs):
print("Before the call")
return func(*args, **kwargs)

wrapper = Wrapper()

functools.update_wrapper(wrapper=wrapper, wrapped=func)

wrapper.__doc__ = DocProperty(func)

return wrapper


@decorate
def some_func(x):
"""Original doc."""
print("Hi from some_func")


print('some_func.__doc__ is {}'.format(some_func.__doc__))
# some_func.__doc__ is <__main__.DocProperty object at 0x7fd134a1d668>

help(some_func)
# Help on Wrapper in module __main__ object:
#
# some_func = class Wrapper(builtins.object)
#  |  Func doc is: 'Original doc.' and now my part...
#  |
#  |  Methods defined here:
#  |
#  |  __call__(self, *args, **kwargs)
#  |
#  |  --
#  |  Data descriptors defined here:
#  |
#  |  __dict__
#  |  dictionary for instance variables (if defined)
#  |
#  |  __weakref__
#  |  list of weak references to the object (if defined)


I tried to decorate the function with a DocProperty, but then hit the wall.
Both __doc__ is not "getted" nor does help() works:

import functools

class DocProperty:
def __init__(self, func):
self.func = func

def __get__(self, instance, owner):
return "Func doc is: {!r} and now my part...".format(self.func.__doc__)

def decorate(func):
def wrapper(*args, **kwargs):
print("Before the call")
return func(*args, **kwargs)

functools.update_wrapper(wrapper=wrapper, wrapped=func)

wrapper.__doc__ = DocProperty(func)

return wrapper

@decorate
def some_func(x):
"""Original doc."""
print("Hi from some_func")

some_func(x=3)
# Before the call
# Hi from some_func

print('some_func.__doc__ is {}'.format(some_func.__doc__))
# some_func.__doc__ is <__main__.DocProperty object at 0x7f0551eea438>

help(some_func)
# Help on function some_func in module __main__:
#
# some_func(x)

I just couldn't figure out how to make the __doc__ attribute of a function
a getter. Is this a bug or am I making a mistake? If it's my mistake, is
there any other way how to dynamically __doc__ a function or am I forced to
set __doc__ of a function at the decoration time?

(If you wonder about the use case: I'd like to dynamically generate the
docstrings when functions are decorated with contracts from icontract
library. Condition functions need to be parsed and re-formatted, so this is
something that should be done on-demand, when the user either wants to see
the help() or when the sphinx documentation is automatically generated. The
docs should not inflict computational overhead during the decoration since
normal course of program operation does not need pretty-printed contracts.)

Thanks for any pointers! Please let me know if you'd like me to compress
one or the other exa

[Python-ideas] Dynamic getting of __doc__ of a function

[Marko Ristin-Kaufmann]

Hi,
I'm working on decorators that have a dynamic __doc__ property computed at
runtime and not at decoration time.

The decorator must return the wrapper as a function and can not return a
callable object with __call__ since the "self" argument would not be
properly passed through the decorator. (Making __call__ a static method
drops implicitly the first argument). If __call__ could be a static method
and get the arguments as-are, I could make __doc__ a property. But this is
not possible,  right?

Assigning a class satisfying the property protocol to the function
wrapper.__doc__ did not work either since the property object is returned
as-is instead of invoking __get__ on it.

Could someone explain a bit -- is this the desired behavior or a bug or an
underspecification?

Is there any way for a callable object to be used as a decorator?

Thanks!

Cheers Marko
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Transpiling contracts

[Marko Ristin-Kaufmann]

Hi Ned,

The idea is to polish a proof-of-concept library and then try to introduce
it into the standard libs eventually.

On Tue, 2 Oct 2018 at 16:57, Ned Batchelder  wrote:

> I'm getting confused: is this still about an idea for Python, or
> development of a third-party library?
>
> --Ned.
>
> On 10/2/18 1:14 AM, Marko Ristin-Kaufmann wrote:
>
> Hi James,
>
> I had another take at it. I wrote it down in the github issue (
> https://github.com/Parquery/icontract/issues/48#issuecomment-426147468):
>
> SLOW=os.environ.get("SOME_ENVIRONMENT_VARIABLE", "") != ""
> class SomeClass:
> def __init__(self)->None:
> self.some_prop = 1984
> def some_func(arg1: List[int], arg2: List[int])->SomeClass:
> with requiring:
> len(arg1) > 3, "some description"
> len(arg2) > 5, ValueError("some format {} {}".format(arg1, arg2))
>
> if SLOW:
> all(elt in arg2 for elt in arg1)
> len(arg2) == len(set(arg2))
>
> result = None  # type: SomeClass
> with ensuring, oldie as O:
> len(arg1) == len(O.arg1)
> result.some_prop < len(arg2)
>
>
> This transpiles to/from:
>
> SLOW=os.environ.get("SOME_ENVIRONMENT_VARIABLE", "") != ""
> class SomeClass:
> def __init__(self)->None:
> self.some_prop = 1984
> @requires(lambda P: len(P.arg1) > 3, "some description")@requires(lambda P: 
> len(P.arg2),
>   error=lambda P: ValueError("some format {} {}".format(P.arg1, 
> P.arg2)))@requires(lambda P: all(elt in P.arg2 for elt in P.arg1), 
> enabled=SLOW)@requires(lambda P: len(arg2) == len(set(arg2)))@ensures(lambda 
> O, P: len(P.arg1) == len(O.arg1))@ensures(lambda P, result: result.some_prop 
> < len(P.arg2))def some_func(arg1: List[int], arg2: List[int])->SomeClass:
> ...
>
>
> 2. What’s the use case for preferring block syntax over lambda syntax? Is
>> the block syntax only better when multiple statements are needed to test a
>> single contract condition?
>>
> Actually no. The condition must be a single statement (specified as a
> boolean expression) regardless of the form (readable or executable).
>
> My idea was that contracts would be a more readable + easier to type and
> group by the respective enabled flags. It should be a 1-to-1 back and forth
> transformation. I didn't aim for any other advantages other than
> readability/easier modification.
>
> The use case I have in mind is not to separate each file into two
> (executable and readable). I'm more thinking of a tool that I can attach to
> key shortcuts in the IDE that let me quickly transform the contracts into
> readable form when I need to read them / make my modifications and then
> convert them back into executable form.
>
> 3. When we do need multiple statements to formally verify/specify, is it
>> usually a better idea to factor out major functionality of the contract
>> testing to an external function? (So other contracts can use the same
>> external function and the intent of the contract is clear trough the
>> function name)
>>
>
> I would say so (i.e. no multi-statement conditions) . Making
> multi-statement conditions would be rather confusing and also harder to put
> into documentation. And it would be hard to implement :)
>
> 4. If this is true, is it a good idea to include a contracts/ folder at
>> the same folder level or a .contracts.py file to list “helper” or
>> verification functions for a contract?
>>
>
> So far, we never had to make a function external to a contract -- it was
> often the signal that something had to be refactored out of the function if
> the condition became complex, but the resulting functions usually either
> stayed in the same file or were more general and had to move to a separate
> module anyhow. Our Python code base has about 95K LOC, I don't know how
> contracts behave on larger code bases or how they would need to be
> structured.
>
> Cheers,
> Marko
>
>
>
> On Mon, 1 Oct 2018 at 17:18, James Lu  wrote:
>
>> Hi Marko,
>>
>> I’m going to refer to the transpiler syntax as “block syntax.”
>>
>> 1. How does Eiffel do formal contracts?
>> 2. What’s the use case for preferring block syntax over lambda syntax? Is
>> the block syntax only better when multiple statements are needed to test a
>> single contract condition?
>> 2. If the last proposition is true, how often do we need multiple
>> statements to test or specify a single contract condition?
>> 3. When we do need multiple statements to formally verify/specify, is it
>> 

Re: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely)

[Marko Ristin-Kaufmann]

Hi Steven (D'Aprano),

Right now, that choice is very restrictive, and I personally don't like
> the look and feel of existing contract libraries. I would give my right
> eye for Cobra-like syntax for contracts in Python:
>
> http://cobra-language.com/trac/cobra/wiki/Contracts
>

You might be interested in the discussion on transpiling contracts back and
forth. See the other thread  "Transpile contracts" and the issue in github:
https://github.com/Parquery/icontract/issues/48
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Transpiling contracts

[Marko Ristin-Kaufmann]

ct is
> callable and treat it as a lambda.
>
> Your approach has better type hinting, but I’m not sure if type hinting
> would be that useful if you’re adding a contract whilst writing the
> function.
>
> James Lu
>
> On Oct 1, 2018, at 1:01 AM, Marko Ristin-Kaufmann 
> wrote:
>
> Hi James,
>
> Regarding the “transpile into Python” syntax with with statements: Can I
>> see an example of this syntax when used in pathlib? I’m a bit worried this
>> syntax is too long and “in the way”, unlike decorators which are before the
>> function body. Or do you mean that both MockP and your syntax should be
>> supported?
>>
>> Would
>>
>> with requiring: assert arg1 < arg2, “message”
>>
>> Be the code you type or the code that’s actually run?
>>
>
> That's the code you would type. Let me make a full example (I'm omitting
> "with contracts" since we actually don't need it).
>
> You would read/write this:
> def some_func(arg1: List[int])->int:
>   with requiring:
> assert len(arg1) > 3, "some description"
>
>   with oldie as O, resultie as result, ensuring:
> if SLOW:
>   O.var1 = sum(arg1)
>   assert result > sum(arg1) > O.var1
>
> assert len(result) > 5
>
> This would run:
> @requires(lambda P: len(P.arg1) > 3, "some description")
> @snapshot(lambda P, var1: sum(P.arg1), enabled=SLOW)
> @ensures(lambda O, P, result: result > sum(arg1) > O.var1, enabled=SLOW)
> @ensures(lambda result: len(result) > 5)
>
> I omitted in the example how to specify a custom exception to avoid
> confusion.
>
> If we decide that transpiler is the way to go, I'd say it's easier to just
> stick with lambdas and allow no mock-based approach in transpilation.
>
> Cheers,
> Marko
>
>
>>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Renaming icontract to pcontract

[Marko Ristin-Kaufmann]

Ok, let's keep icontract as it was already referenced a couple of times at
least on this list.

I'll rename the pypi package to icontract2 and the module as well as soon
as I change the interface.

On Mon, 1 Oct 2018 at 20:43, James Lu  wrote:

> I think the confusion would be minimal and not worth the hassle of
> renaming to pcontract. People would just say “Python icontract” when it’s
> ambiguous.
>
> Sent from my iPhone
>
> On Oct 1, 2018, at 3:28 AM, Marko Ristin-Kaufmann 
> wrote:
>
> Hi Cameron,
> A nerdy way to make it sound like a sentence: "I contract that ...".
>
> Pcontract would stand for python contract. Pycontract is already taken.
>
> Cheers,
> Marko
>
> Le lun. 1 oct. 2018 à 09:15, Cameron Simpson  a écrit :
>
>> On 01Oct2018 07:25, Marko Ristin-Kaufmann  wrote:
>> >I'd like to rename icontract into pcontract to avoid name conflict with
>> >java's icontract library.
>> >
>> >Do you have any better suggestion?
>>
>> No, sounds ok to me. What was the "i" for in the old name?
>>
>> Cheers,
>> Cameron Simpson 
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely)

[Marko Ristin-Kaufmann]

Hi Steve (Turnbull),

It sounds to me like the proof of the pudding for Python will be
> annotating the ABC module with contracts.  If that turns out to be a
> mostly fruitless exercise, it's hard to see how "real" contracts
> (vs. using asserts to "program in contracting style") are an important
> feature for Python in general.
>

Do you mean annotating an abstract class with contracts or annotating the
actual functions in abc module?

If you meant abstract classes in general, we do that for quite some time in
our code base and had no problems so far. icontract already supports
inheritance of contracts (as well as weakening/strengthening of the
contracts).

Inheriting of contracts also proved useful when you define an interface as
a group of functions together with the contracts. You define the functions
as an ABC with purely static methods (no state) and specify the contracts
in that abstract class. The concrete classes implement the static
functions, but don't need to repeat the contracts since they are inherited
from the abstract class. I can copy/paste from our code base if you are
interested.

Cheers,
Marko

On Mon, 1 Oct 2018 at 15:09, Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Elazar writes:
>  > On Sun, Sep 30, 2018, 15:12 Stephen J. Turnbull <
>  > turnbull.stephen...@u.tsukuba.ac.jp> wrote:
>
>  > > What does "inherited" mean?  Just that methods that are not overridden
>  > > retain their contracts?
>  >
>  > Contracts are attached to interfaces, not to specifications. So
>  > when you have abstract base class, it defines contracts, and
>  > implementing classes must adhere to these contracts - the can only
>  > strengthen it, not weaken it.
>
> Stated in words, it sounds very reasonable.
>
> Thinking about the implications for writing Python code, it sounds
> very constraining in the context of duck-typing.  Or, to give a
> contrapositive example, you can write few, if any, contracts for
> dunders beyond their signatures.  For example, len(x) + len(y) ==
> len(x + y) is not an invariant over the classes that define __len__
> (eg, set and dict, taking the "x + y" loosely for them).  Of course if
> you restrict to Sequence, you can impose that invariant.  The question
> is the balance.
>
> It sounds to me like the proof of the pudding for Python will be
> annotating the ABC module with contracts.  If that turns out to be a
> mostly fruitless exercise, it's hard to see how "real" contracts
> (vs. using asserts to "program in contracting style") are an important
> feature for Python in general.
>
> That said, given the passion of the proponents, if we can come up with
> an attractive style for implementing contracts without a language
> change, I'd be +0 at least for adding a module to the stdlib.  But
> this is a Python 3.9 project, given the state of the art.
>
>  > This is precisely like with types, since types are contracts (and
>  > vice versa, in a way).
>
> Not Python types (== classes), though.  As a constraint on behavior,
> subclassing is purely nominal, although Python deliberately makes it
> tedious to turn a subclass of str into a clone of int.
>
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Renaming icontract to pcontract

[Marko Ristin-Kaufmann]

Hi Cameron,
A nerdy way to make it sound like a sentence: "I contract that ...".

Pcontract would stand for python contract. Pycontract is already taken.

Cheers,
Marko

Le lun. 1 oct. 2018 à 09:15, Cameron Simpson  a écrit :

> On 01Oct2018 07:25, Marko Ristin-Kaufmann  wrote:
> >I'd like to rename icontract into pcontract to avoid name conflict with
> >java's icontract library.
> >
> >Do you have any better suggestion?
>
> No, sounds ok to me. What was the "i" for in the old name?
>
> Cheers,
> Cameron Simpson 
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Renaming icontract to pcontract

[Marko Ristin-Kaufmann]

Hi,
I'd like to rename icontract into pcontract to avoid name conflict with
java's icontract library.

Do you have any better suggestion?

Thanks!
Marko
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Transpiling contracts

[Marko Ristin-Kaufmann]

Hi James,

Regarding the “transpile into Python” syntax with with statements: Can I
> see an example of this syntax when used in pathlib? I’m a bit worried this
> syntax is too long and “in the way”, unlike decorators which are before the
> function body. Or do you mean that both MockP and your syntax should be
> supported?
>
> Would
>
> with requiring: assert arg1 < arg2, “message”
>
> Be the code you type or the code that’s actually run?
>

That's the code you would type. Let me make a full example (I'm omitting
"with contracts" since we actually don't need it).

You would read/write this:
def some_func(arg1: List[int])->int:
  with requiring:
assert len(arg1) > 3, "some description"

  with oldie as O, resultie as result, ensuring:
if SLOW:
  O.var1 = sum(arg1)
  assert result > sum(arg1) > O.var1

assert len(result) > 5

This would run:
@requires(lambda P: len(P.arg1) > 3, "some description")
@snapshot(lambda P, var1: sum(P.arg1), enabled=SLOW)
@ensures(lambda O, P, result: result > sum(arg1) > O.var1, enabled=SLOW)
@ensures(lambda result: len(result) > 5)

I omitted in the example how to specify a custom exception to avoid
confusion.

If we decide that transpiler is the way to go, I'd say it's easier to just
stick with lambdas and allow no mock-based approach in transpilation.

Cheers,
Marko


>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

Hi James,
I agree. Having a snapshot decorator per variable also has the advantage
that we can add parameters to the decorator (e.g., snapshots also need
"enabled" and "description" argument).

How should we distinguish the two approaches? (I suppose you would also
apply them to requests, ensures and invariant decorator, right?) Is there a
terminology? Lambda-based conditions might be the name for the conventional
approach with @requests(lambda P: ...). What should we term the approach
you proposed?

Re naming: @requests(lambda P: P.arg1 < P.arg2) and @requests_that(P.arg1 <
P.arg2)?

Le dim. 30 sept. 2018 à 22:07, James Lu  a écrit :

> Hi Marko,
>
> I just found the time to reply to these.
> > I reread the proposal with MockP. I still don't get the details, but if
> I think I understand the basic idea. You put a placeholder and whenever one
> of its methods is called (including dunders), you record it and finally
> assemble an AST and compile a lambda function to be executed at actual call
> later.
> Precisely.
>
>
> > But that would still fail if you want to have:
> > @snapshot(var1=some_func(MockP.arg1, MockP.arg2))
> > , right? Or there is a way to record that?
>
> This would still fail. You would record it like this:
> @snapshot(var1=thunk(some_func)(MockP.arg1, MockP.arg2))
>
> thunk stores the function for later and produces another MockP object that
> listens for __call__.
>
> By the way, MockP is the class while P is a virgin instance of MockP.
> MockP instances are immutable, so any operation on a MockP instance creates
> a new object or MockP instance.
>
> I’m also beginning to lean towards
> @snapshot(var1=...)
> @snapshot(var2=...)
>
> I suspect this would deal better with VCS.
>
> This syntax does have a a nice visual alignment. I’m not entirely sure
> what kind of indentation PEP 8 recommends and editors give, so the point
> may be moot if the natural indentation also gives the same visual alignment.
>
> Though both should be supported so the best syntax may win.
>
> James Lu
>
>
> > On Sep 29, 2018, at 3:22 PM, Marko Ristin-Kaufmann <
> marko.ris...@gmail.com> wrote:
> >
> > I reread the proposal with MockP. I still don't get the details, but if
> I think I understand the basic idea. You put a placeholder and whenever one
> of its methods is called (including dunders), you record it and finally
> assemble an AST and compile a lambda function to be executed at actual call
> later.
> >
> > But that would still fail if you want to have:
> > @snapshot(var1=some_func(MockP.arg1, MockP.arg2))
> > , right? Or there is a way to record that?
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

Hi James,
Sure, let's do that! I'm fine with emails, too. I needed only a bit of
structure since we lumped so many issues together, but maybe it's just my
OCD :)

Le lun. 1 oct. 2018 à 04:05, James Lu  a écrit :

> Hi Marko,
>
> Regarding switching over to GitHub issues:
> * I copy-pasted the MockP original code to GitHub issues.
> * There's a clunky way to view the discussion at
> https://mail.python.org/pipermail/python-ideas/2018-September/subject.html#start
> .
> * The less clunky way to view the discussion is to subscribe to the
> mailing list and use Gmail to move all the messages from python-ideas to
> python ideas list and all the messages from the discussions we have to a
> "contracts" label and view the discussion with your email client.
> * A week earlier I didn't think I'd be saying this, but I like email for
> discussion better. It works on mobile and I can send messages offline, and
> I send 90% of my messages on my phone and when I'm offline. Unless you know
> an alternative (WhatsApp, maybe?) that fits my use cases off the top of
> your head, I think we should stick to email.
> * My proposal: We split the discussion into a new email thread, we keep
> the latest agreed upon proposal on GitHub issues.
>
> On Sun, Sep 30, 2018 at 4:32 PM Marko Ristin-Kaufmann <
> marko.ris...@gmail.com> wrote:
>
>> Hi James,
>> (I'm just about to go to sleep, so I'll answer the other messages
>> tomorrow.)
>>
>> Should we keep some kind of document to keep track of all the different
>>> proposals? I’m thinking an editable document like HackMD where we can label
>>> all the different ideas to keep them straight in our head.
>>>
>>
>> I thought github issues would be a suitable place for that:
>> https://github.com/Parquery/icontract/issues
>>
>> It reads a bit easier as a discussion rather than a single document -- if
>> anybody else needs to follow. What do you think?
>>
>> On Sun, 30 Sep 2018 at 22:07, James Lu  wrote:
>>
>>> Hi Marko,
>>>
>>> Going back to your proposal on repeating lambda P as a convention.
>>>
>>> I do find
>>>
>>> @snapshot(some_identifier=P -> P.self(P.arg1),
>>> some_identifier2=P -> P.arg1 + P.arg2)
>>>
>>> acceptable.
>>>
>>> Should we keep some kind of document to keep track of all the different
>>> proposals? I’m thinking an editable document like HackMD where we can label
>>> all the different ideas to keep them straight in our head.
>>>
>>> James Lu
>>
>>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

Hi James,
(I'm just about to go to sleep, so I'll answer the other messages tomorrow.)

Should we keep some kind of document to keep track of all the different
> proposals? I’m thinking an editable document like HackMD where we can label
> all the different ideas to keep them straight in our head.
>

I thought github issues would be a suitable place for that:
https://github.com/Parquery/icontract/issues

It reads a bit easier as a discussion rather than a single document -- if
anybody else needs to follow. What do you think?

On Sun, 30 Sep 2018 at 22:07, James Lu  wrote:

> Hi Marko,
>
> Going back to your proposal on repeating lambda P as a convention.
>
> I do find
>
> @snapshot(some_identifier=P -> P.self(P.arg1),
> some_identifier2=P -> P.arg1 + P.arg2)
>
> acceptable.
>
> Should we keep some kind of document to keep track of all the different
> proposals? I’m thinking an editable document like HackMD where we can label
> all the different ideas to keep them straight in our head.
>
> James Lu
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Simplicity of C (was why is design-by-contracts not widely)

[Marko Ristin-Kaufmann]

Hi Cameron,

Just a word of caution: I made a mistake and badly designed interface to
icontract. It will change in the near future from:

@post(lambda arg1, arg2, result: arg1 < result < arg2)

most probably to:

@ensures(lambda P:  P.arg1 < result < P.arg2)

This avoids any name conflicts with "result" if it's in the arguments and
also many conflicts with web frameworks which frequently use "post". We
will also add snapshotting before the function execution:
@snapshot(lambda P, var2: set(arg2))
@ensures(lambda O, P: P.arg1 < result and result in O.var2)

so that postcondition can deal with state transitions. There are also some
other approaches in discussion.

The library name will also need to change. When I started developing it, I
was not aware of Java icontract library. It will be probably renamed to
"pcontract" or any other suggested better name :)

Please see the github issues for more details and current discussions:
https://github.com/Parquery/icontract/issues

On Sun, 30 Sep 2018 at 06:44, Cameron Simpson  wrote:

> On 30Sep2018 12:17, Chris Angelico  wrote:
> >At the moment, I'm seeing decorator-based contracts as a clunky
> >version of unit tests. We already have "inline unit testing" - it's
> >called doctest - and I haven't seen anything pinned down as "hey, this
> >is what it'd take to make contracts more viable". Certainly nothing
> >that couldn't be done as a third-party package. But I'm still open to
> >being swayed on that point.
>
> Decorator based contracts are very little like clunky unit tests to me.
> I'm
> basing my opinion on the icontracts pip package, which I'm going to start
> using.
>
> In case you've been looking at something different, it provides a small
> number
> of decorators including @pre(test-function) and @post(test-function) and
> the
> class invariant decorator @inv, with good error messages for violations.
>
> They are _functionally_ like putting assertions in your code at the start
> and
> end of your functions, but have some advantages:
>
> - they're exposed, not buried inside the function, where they're easy to
> see
>   and can be considered as contracts
>
> - they run on _every_ function call, not just during your testing, and get
>   turned off just like assertions do: when you run Python with the -O
>   (optimise) option. (There's some more tuning available too.)
>
> - the assertions make qualitative statements about the object/parameter
> state
>   in the form "the state is consistent if these things apply";
>   tests tend to say "here's a situation, do these things and examine these
>   results". You need to invent the situations and the results, rather than
>   making general statements about the purpose and functional semantics of
> the
>   class.
>
> They're different to both unit tests _and_ doctests because they get
> exercised
> during normal code execution. Both unit tests and doctests run _only_
> during
> your test phase, with only whatever test scenarios you have devised.
>
> The difficulty with unit tests and doctests (both of which I use) and also
> integration tests is making something small enough to run but big/wide
> enough
> to cover all the relevant cases. They _do not_ run against all your real
> world
> data. It can be quite hard to apply them to your real world data.
>
> Also, all the @pre/@post/@inv stuff will run _during_ your unit tests and
> doctests as well, so they get included in your test regime for free.
>
> I've got a few classes which have a selftest method whose purpose is to
> confirm
> correctness of the instance state, and I call that from a few methods at
> start
> and end, particularly those for which unit tests have been hard to write
> or I
> know are inadequately covered (and probably never will be because devising
> a
> sufficient test case is impractical, especially for hard to envisage
> corner
> cases).
>
> The icontracts module will be very helpful to me here: I can pull out the
> self-test function as the class invariant, and make a bunch of @pre/@post
> assertions corresponding the the method semantic definition.
>
> The flip side of this is that there's no case for language changes in what
> I
> say above: the decorators look pretty good to my eye.
>
> Cheers,
> Cameron Simpson 
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi,

I compiled a couple of issues on github to provide a more structured ground
for discussions on icontract features:
https://github.com/Parquery/icontract/issues (@David Maertz: I also
included the issue with automatically generated __doc__ in case you are
still interested in it).

Cheers,
Marko

On Sat, 29 Sep 2018 at 17:27, Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Steven D'Aprano writes:
>
>  > put (x: ELEMENT; key: STRING) is
>  >  -- Insert x so that it will be retrievable through key.
>  >  require
>  >  count <= capacity
>  >  not key.empty
>  >  do
>  >  ... Some insertion algorithm ...
>  >  ensure
>  >  has (x)
>  >  item (key) = x
>  >  count = old count + 1
>  >  end
>  >
>  > Two pre-conditions, and three post-conditions. That's hardly
>  > complex.
>
> You can already do this:
>
> def put(self, x: Element, key: str) -> None:
> """Insert x so that it will be retrievable through key."""
>
> # CHECKING PRECONDITIONS
> _old_count = self.count
> assert self.count <= self.capacity,
> assert key
>
> # IMPLEMENTATION
> ... some assertion algorithm ...
>
> # CHECKING POSTCONDITIONS
> assert x in self
> assert self[key] == x
> assert self.count == _old_count
>
> return
>
> I don't see a big advantage to having syntax, unless the syntax allows
> you to do things like turn off "expensive" contracts only.  Granted,
> you save a little bit of typing and eye movement (you can omit
> "assert" and have syntax instead of an assignment for checking
> postconditions dependent on initial state).
>
> A document generator can look for the special comments (as with
> encoding cookies), and suck in all the asserts following until a
> non-assert line of code (or the next special comment).  The
> assignments will need special handling, an additional special comment
> or something.  With PEP 572, I think you could even do this:
>
> assert ((_old_count := self.count),)
>
> to get the benefit of python -O here.
>
>  > If I were writing this in Python, I'd write something like this:
>  >
>  > def put(self, x, key):
>  > """Insert x so that it will be retrievable through key."""
>  > # Input checks are pre-conditions!
>  > if self.count > capacity:
>  > raise DatabaseFullError
>  > if not key:
>  > raise ValueError
>  > # .. Some insertion algorithm ...
>
> But this is quite different, as I understand it.  Nothing I've seen in
> the discussion so far suggests that a contract violation allows
> raising differentiated exceptions, and it seems very unlikely from the
> syntax in your example above.  I could easily see both of these errors
> being retryable:
>
> for _ in range(3):
> try:
> db.put(x, key)
> except DatabaseFullError:
> db.resize(expansion_factor=1.5)
> db.put(x, key)
> except ValueError:
> db.put(x, alternative_key)
>
>  > and then stick the post-conditions in a unit test, usually in a
>  > completely different file:
>
> If you like the contract-writing style, why would you do either of
> these instead of something like the code I wrote above?
>
>  > So what's wrong with the status quo?
>  >
>  > - The pre-condition checks are embedded right there in the
>  >   method implementation, mixing up the core algorithm with the
>  >   associated error checking.
>
> You don't need syntax to separate them, you can use a convention, as I
> did above.
>
>  > - Which in turn makes it hard to distinguish the checks from
>  >   the implementation, and impossible to do so automatically.
>
> sed can do it, why can't we?
>
>  > - Half of the checks are very far away, in a separate file,
>  >   assuming I even remembered or bothered to write the test.
>
> That was your choice.  There's nothing about the assert statement that
> says you're not allowed to use it at the end of a definition.
>
>  > - The post-conditions aren't checked unless I run my test suite, and
>  >   then they only check the canned input in the test suite.
>
> Ditto.
>
>  > - The pre-conditions can't be easily disabled in production.
>
> What's so hard about python -O?
>
>  > - No class invariants.
>
> Examples?
>
>  > - Inheritance is not handled correctly.
>
> Examples?  Mixins and classes with additional functionality should
> work fine AFAICS.  I guess you'd have to write the contracts in each
> subclass of an abstract class, which is definitely a minus for some of
> the contracts.  But I don't see offhand why you would expect that the
> full contract of a method of a parent class would typically make sense
> without change for an overriding implementation, and might not make
> sense for a class with restricted functionality.
>
>  > The status quo is all so very ad-hoc and messy. Design By Contract
>  > syntax would 

Re: [Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

Hi James,

I copy/pasted the discussion re the readability tool to an issue on github:
https://github.com/Parquery/icontract/issues/48

Would you mind opening a separate issue and copy/pasting what you find
relevant re MockP approach in a separate issue?

I think it's time to fork the issues and have separate threads with code
highlighting etc. Is that OK with you?

Cheers,
Marko

On Sat, 29 Sep 2018 at 21:22, Marko Ristin-Kaufmann 
wrote:

> Hi James,
> I reread the proposal with MockP. I still don't get the details, but if I
> think I understand the basic idea. You put a placeholder and whenever one
> of its methods is called (including dunders), you record it and finally
> assemble an AST and compile a lambda function to be executed at actual call
> later.
>
> But that would still fail if you want to have:
> @snapshot(var1=some_func(MockP.arg1, MockP.arg2))
> , right? Or there is a way to record that?
>
> Cheers,
> Marko
>
> Le sam. 29 sept. 2018 à 00:35, James Lu  a écrit :
>
>> I am fine with your proposed syntax. It’s certainly lucid. Perhaps it
>> would be a good idea to get people accustomed to “non-magic” syntax.
>>
>> I still have a feeling that most developers would like to store the state
>> in many different custom ways.
>>
>> Please explain. (Expressions like thunk(all)(a == b for a, b in
>> P.arg.meth()) would be valid.)
>>
>> I'm thinking mostly about all the edge cases which we would not be able
>> to cover (and how complex that would be to cover them).
>>
>>
>> Except for a > b > c being one flat expression with 5 members, it seems
>> fairly easy to recreate an AST, which can then be compiled down to a code
>> object. The code object can be fun with a custom “locals()”
>>
>> Below is my concept code for such a P object.
>>
>> from ast import *
>>
>> # not done: enforce Singleton property on EmptySymbolType
>>
>> class EmptySymbolType(object): ...
>>
>> EmptySymbol = EmptySymbolType() # empty symbols are placeholders
>>
>> class MockP(object):
>>
>> # "^" is xor
>>
>> @icontract.pre(lambda symbol, astnode: (symbol is None) ^ (astnode is
>> None))
>>
>> def __init__(self, symbol=None, value=EmptySymbol, astnode=None,
>> initsymtable=(,)):
>>
>> self.symtable = dict(initsymtable)
>>
>> if symbol:
>>
>> self.expr = Expr(value=Name(id=symbol, ctx=Load()))
>>
>> self.symtable = {symbol: value}
>>
>> else:
>>
>> self.expr = astnode
>>
>> self.frozen = False
>>
>> def __add__(self, other):
>>
>> wrapped = MockP.wrap_value(other)
>>
>> return MockP(astnode=Expr(value=BinOp(self.expr, Add(),
>> wrapped.expr), initsymtable={**self.symtable, **wrapped.symtable})
>>
>> def compile(self): ...
>>
>> def freeze(self):
>>
>> # frozen objects wouldn’t have an overrided getattr, allowing for
>> icontract to manipulate the MockP object using its public interface
>>
>> self.frozen = True
>>
>> @classmethod
>>
>> def wrap_value(cls, obj):
>>
>># create a MockP object from a value. Generate a random identifier
>> and set that as the key in symtable, the AST node is the name of that
>> identifier, retrieving its value through simple expression evaluation.
>>
>>...
>>
>>
>> thunk = MockP.wrap_value
>>
>> P = MockP('P')
>>
>> # elsewhere: ensure P is only accessed via valid “dot attribute access”
>> inside @snapshot so contracts fail early, or don’t and allow Magic like
>> __dict__ to occur on P.
>>
>> On Sep 27, 2018, at 9:49 PM, Marko Ristin-Kaufmann <
>> marko.ris...@gmail.com> wrote:
>>
>> Hi James,
>>
>> I still have a feeling that most developers would like to store the state
>> in many different custom ways. I see also thunk and snapshot with wrapper
>> objects to be much more complicated to implement and maintain; I'm thinking
>> mostly about all the edge cases which we would not be able to cover (and
>> how complex that would be to cover them). Then the linters need also to
>> work around such wrappers... It might also scare users off since it looks
>> like too much magic. Another concern I also have is that it's probably very
>> hard to integrate these wrappers with mypy later -- but I don't really have
>> a clue about that, only my gut feeling?
>>
>> What about we accepted to repeat "

Re: [Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

Hi James,
I reread the proposal with MockP. I still don't get the details, but if I
think I understand the basic idea. You put a placeholder and whenever one
of its methods is called (including dunders), you record it and finally
assemble an AST and compile a lambda function to be executed at actual call
later.

But that would still fail if you want to have:
@snapshot(var1=some_func(MockP.arg1, MockP.arg2))
, right? Or there is a way to record that?

Cheers,
Marko

Le sam. 29 sept. 2018 à 00:35, James Lu  a écrit :

> I am fine with your proposed syntax. It’s certainly lucid. Perhaps it
> would be a good idea to get people accustomed to “non-magic” syntax.
>
> I still have a feeling that most developers would like to store the state
> in many different custom ways.
>
> Please explain. (Expressions like thunk(all)(a == b for a, b in
> P.arg.meth()) would be valid.)
>
> I'm thinking mostly about all the edge cases which we would not be able to
> cover (and how complex that would be to cover them).
>
>
> Except for a > b > c being one flat expression with 5 members, it seems
> fairly easy to recreate an AST, which can then be compiled down to a code
> object. The code object can be fun with a custom “locals()”
>
> Below is my concept code for such a P object.
>
> from ast import *
>
> # not done: enforce Singleton property on EmptySymbolType
>
> class EmptySymbolType(object): ...
>
> EmptySymbol = EmptySymbolType() # empty symbols are placeholders
>
> class MockP(object):
>
> # "^" is xor
>
> @icontract.pre(lambda symbol, astnode: (symbol is None) ^ (astnode is
> None))
>
> def __init__(self, symbol=None, value=EmptySymbol, astnode=None,
> initsymtable=(,)):
>
> self.symtable = dict(initsymtable)
>
> if symbol:
>
> self.expr = Expr(value=Name(id=symbol, ctx=Load()))
>
> self.symtable = {symbol: value}
>
> else:
>
> self.expr = astnode
>
> self.frozen = False
>
> def __add__(self, other):
>
> wrapped = MockP.wrap_value(other)
>
> return MockP(astnode=Expr(value=BinOp(self.expr, Add(),
> wrapped.expr), initsymtable={**self.symtable, **wrapped.symtable})
>
> def compile(self): ...
>
> def freeze(self):
>
> # frozen objects wouldn’t have an overrided getattr, allowing for
> icontract to manipulate the MockP object using its public interface
>
> self.frozen = True
>
> @classmethod
>
> def wrap_value(cls, obj):
>
># create a MockP object from a value. Generate a random identifier
> and set that as the key in symtable, the AST node is the name of that
> identifier, retrieving its value through simple expression evaluation.
>
>...
>
>
> thunk = MockP.wrap_value
>
> P = MockP('P')
>
> # elsewhere: ensure P is only accessed via valid “dot attribute access”
> inside @snapshot so contracts fail early, or don’t and allow Magic like
> __dict__ to occur on P.
>
> On Sep 27, 2018, at 9:49 PM, Marko Ristin-Kaufmann 
> wrote:
>
> Hi James,
>
> I still have a feeling that most developers would like to store the state
> in many different custom ways. I see also thunk and snapshot with wrapper
> objects to be much more complicated to implement and maintain; I'm thinking
> mostly about all the edge cases which we would not be able to cover (and
> how complex that would be to cover them). Then the linters need also to
> work around such wrappers... It might also scare users off since it looks
> like too much magic. Another concern I also have is that it's probably very
> hard to integrate these wrappers with mypy later -- but I don't really have
> a clue about that, only my gut feeling?
>
> What about we accepted to repeat "lambda P, " prefix, and have something
> like this:
>
> @snapshot(
>   lambda P, some_name: len(P.some_property),
>   lambda P, another_name: hash(P.another_property)
> )
>
> It's not too verbose for me and you can still explain in three-four
> sentences what happens below the hub in the library's docs.  A
> pycharm/pydev/vim/emacs plugins could hide the verbose parts.
>
> I performed a small experiment to test how this solution plays with pylint
> and it seems OK that arguments are not used in lambdas.
>
> Cheers,
> Marko
>
>
> On Thu, 27 Sep 2018 at 12:27, James Lu  wrote:
>
>> Why couldn’t we record the operations  done to a special object and
>> replay them?
>>
>> Actually, I think there is probably no way around a decorator that
>>> captures/snapshots the data before the function call with a lambda (or even
>>> a separate fun

Re: [Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

P.p.s. to raise a custom exception:

if not (arg1 < S.var1 < arg2):
"Some description"
raise SomeException(arg1, S.var1, arg2)

The converter enforces that only "if not" statement is allowed, only a
string description (optional) followed by a raise in the body of
if-statement.

This is later at back-conversion to python easy to transform into a lambda.

Cheers,
Marko




Le sam. 29 sept. 2018 à 20:56, Marko Ristin-Kaufmann 
a écrit :

> Hi James,
> Just a PS to the previous syntax:
>
> with contracts:
> with preconditions:
> assert arg1 < arg2
>
> with snapshot as S:
> S.var1 = some_func(arg1)
> with postconditions, \
>  result:
>  # result would be annotated with "# type:" if return type
> is annotated.
>   assert arg1 < S.var1 < arg2
>
> For classes:
> class SomeClass:
> with invariants,
> selfie as self: # type: SomeClass
>  assert 0 < self.x < sqrt(self.x)
>
> The advantage: no variable shadowing, valid python code, autocomplete
> works in Pycharm, even mypy could be made to work. "With contracts" makes
> it easier and less error prone to group preconditions and postconditions.
> The converter would check that there is no "with contracts" in the body of
> the function except in the first statement and the same for class
> invariants.
>
> icontract.dummies would provide these dummy context managers (all of them
> would raise exceptions on enter so that the code can not run by accident).
> The converter would add/remove these imports automatically.
>
> Cheers,
> Marko
>
>
> Le sam. 29 sept. 2018 à 17:55, Marko Ristin-Kaufmann <
> marko.ris...@gmail.com> a écrit :
>
>> Hi James,
>> What about a tool that we discussed, to convert contracts back and forth
>> to readable form on IDe save/load with the following syntax:
>>
>> def some_func(arg1:int, arg2:int)-> int:
>> # typing on the phone so no indent
>> With requiring:
>> Assert arg1 < arg2, "some message"
>> With snapshotting:
>> Var1= some_func(arg1)
>>
>> With ensuring:
>>  If some_enabling_condition:
>> Assert arg1 + arg2 < var1
>>
>> If no snapshot, with ensuring is dedented. Only simple assignments
>> allowed in snapshots,  only asserts and ifs allowed in require/ensure
>> blocks. Result is reserved for the result of the function.
>>
>> No statements allowed in require/ensure.
>>
>> The same with class invariants.
>>
>> Works with ast and autocomplete in pycharm.
>>
>> Sorry for the hasty message :)
>> Marko
>>
>>
>>
>> Le sam. 29 sept. 2018 à 07:36, Marko Ristin-Kaufmann <
>> marko.ris...@gmail.com> a écrit :
>>
>>> Hi James,
>>> I'm a bit short on time today, and would need some more time and
>>> attention to understand the proposal you wrote. I'll try to come back to
>>> you tomorrow.
>>>
>>> In any case, I need to refactor icontract's decorators to use conditions
>>> like lambda P: and lambda P, result: first before adding snapshot
>>> functionality.
>>>
>>> What about having @snapshot_with and @snapshot? @Snapshot_with does what
>>> you propose and @snapshot expects a lambda P, identifier: ?
>>>
>>> After the refactoring, maybe the same could be done for defining
>>> contracts as well? (Requires and requires_that?)
>>>
>>> If the documentation is clear, I'd expect the user to be able to
>>> distinguish the two. The first approach is shorter, and uses magic, but
>>> fails in some rare situations. The other method is more verbose, but always
>>> works.
>>>
>>> Cheers,
>>> Marko
>>>
>>> Le sam. 29 sept. 2018 à 00:35, James Lu  a écrit :
>>>
>>>> I am fine with your proposed syntax. It’s certainly lucid. Perhaps it
>>>> would be a good idea to get people accustomed to “non-magic” syntax.
>>>>
>>>> I still have a feeling that most developers would like to store the
>>>> state in many different custom ways.
>>>>
>>>> Please explain. (Expressions like thunk(all)(a == b for a, b in
>>>> P.arg.meth()) would be valid.)
>>>>
>>>> I'm thinking mostly about all the edge cases which we would not be able
>>>> to cover (and how complex that would be to cover them).
>>>>
>>>>
>>>> Except for a > b > c being one flat expres

Re: [Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

Hi James,
What about a tool that we discussed, to convert contracts back and forth to
readable form on IDe save/load with the following syntax:

def some_func(arg1:int, arg2:int)-> int:
# typing on the phone so no indent
With requiring:
Assert arg1 < arg2, "some message"
With snapshotting:
Var1= some_func(arg1)

With ensuring:
 If some_enabling_condition:
Assert arg1 + arg2 < var1

If no snapshot, with ensuring is dedented. Only simple assignments allowed
in snapshots,  only asserts and ifs allowed in require/ensure blocks.
Result is reserved for the result of the function.

No statements allowed in require/ensure.

The same with class invariants.

Works with ast and autocomplete in pycharm.

Sorry for the hasty message :)
Marko



Le sam. 29 sept. 2018 à 07:36, Marko Ristin-Kaufmann 
a écrit :

> Hi James,
> I'm a bit short on time today, and would need some more time and attention
> to understand the proposal you wrote. I'll try to come back to you
> tomorrow.
>
> In any case, I need to refactor icontract's decorators to use conditions
> like lambda P: and lambda P, result: first before adding snapshot
> functionality.
>
> What about having @snapshot_with and @snapshot? @Snapshot_with does what
> you propose and @snapshot expects a lambda P, identifier: ?
>
> After the refactoring, maybe the same could be done for defining contracts
> as well? (Requires and requires_that?)
>
> If the documentation is clear, I'd expect the user to be able to
> distinguish the two. The first approach is shorter, and uses magic, but
> fails in some rare situations. The other method is more verbose, but always
> works.
>
> Cheers,
> Marko
>
> Le sam. 29 sept. 2018 à 00:35, James Lu  a écrit :
>
>> I am fine with your proposed syntax. It’s certainly lucid. Perhaps it
>> would be a good idea to get people accustomed to “non-magic” syntax.
>>
>> I still have a feeling that most developers would like to store the state
>> in many different custom ways.
>>
>> Please explain. (Expressions like thunk(all)(a == b for a, b in
>> P.arg.meth()) would be valid.)
>>
>> I'm thinking mostly about all the edge cases which we would not be able
>> to cover (and how complex that would be to cover them).
>>
>>
>> Except for a > b > c being one flat expression with 5 members, it seems
>> fairly easy to recreate an AST, which can then be compiled down to a code
>> object. The code object can be fun with a custom “locals()”
>>
>> Below is my concept code for such a P object.
>>
>> from ast import *
>>
>> # not done: enforce Singleton property on EmptySymbolType
>>
>> class EmptySymbolType(object): ...
>>
>> EmptySymbol = EmptySymbolType() # empty symbols are placeholders
>>
>> class MockP(object):
>>
>> # "^" is xor
>>
>> @icontract.pre(lambda symbol, astnode: (symbol is None) ^ (astnode is
>> None))
>>
>> def __init__(self, symbol=None, value=EmptySymbol, astnode=None,
>> initsymtable=(,)):
>>
>> self.symtable = dict(initsymtable)
>>
>> if symbol:
>>
>> self.expr = Expr(value=Name(id=symbol, ctx=Load()))
>>
>> self.symtable = {symbol: value}
>>
>> else:
>>
>> self.expr = astnode
>>
>> self.frozen = False
>>
>> def __add__(self, other):
>>
>> wrapped = MockP.wrap_value(other)
>>
>> return MockP(astnode=Expr(value=BinOp(self.expr, Add(),
>> wrapped.expr), initsymtable={**self.symtable, **wrapped.symtable})
>>
>> def compile(self): ...
>>
>> def freeze(self):
>>
>> # frozen objects wouldn’t have an overrided getattr, allowing for
>> icontract to manipulate the MockP object using its public interface
>>
>> self.frozen = True
>>
>> @classmethod
>>
>> def wrap_value(cls, obj):
>>
>># create a MockP object from a value. Generate a random identifier
>> and set that as the key in symtable, the AST node is the name of that
>> identifier, retrieving its value through simple expression evaluation.
>>
>>...
>>
>>
>> thunk = MockP.wrap_value
>>
>> P = MockP('P')
>>
>> # elsewhere: ensure P is only accessed via valid “dot attribute access”
>> inside @snapshot so contracts fail early, or don’t and allow Magic like
>> __dict__ to occur on P.
>>
>> On Sep 27, 2018, at 9:49 PM, Marko Ristin-Kaufmann <
>> marko.ris...@gmail.com> wrote:
>>
>> Hi

Re: [Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

Hi James,
I'm a bit short on time today, and would need some more time and attention
to understand the proposal you wrote. I'll try to come back to you
tomorrow.

In any case, I need to refactor icontract's decorators to use conditions
like lambda P: and lambda P, result: first before adding snapshot
functionality.

What about having @snapshot_with and @snapshot? @Snapshot_with does what
you propose and @snapshot expects a lambda P, identifier: ?

After the refactoring, maybe the same could be done for defining contracts
as well? (Requires and requires_that?)

If the documentation is clear, I'd expect the user to be able to
distinguish the two. The first approach is shorter, and uses magic, but
fails in some rare situations. The other method is more verbose, but always
works.

Cheers,
Marko

Le sam. 29 sept. 2018 à 00:35, James Lu  a écrit :

> I am fine with your proposed syntax. It’s certainly lucid. Perhaps it
> would be a good idea to get people accustomed to “non-magic” syntax.
>
> I still have a feeling that most developers would like to store the state
> in many different custom ways.
>
> Please explain. (Expressions like thunk(all)(a == b for a, b in
> P.arg.meth()) would be valid.)
>
> I'm thinking mostly about all the edge cases which we would not be able to
> cover (and how complex that would be to cover them).
>
>
> Except for a > b > c being one flat expression with 5 members, it seems
> fairly easy to recreate an AST, which can then be compiled down to a code
> object. The code object can be fun with a custom “locals()”
>
> Below is my concept code for such a P object.
>
> from ast import *
>
> # not done: enforce Singleton property on EmptySymbolType
>
> class EmptySymbolType(object): ...
>
> EmptySymbol = EmptySymbolType() # empty symbols are placeholders
>
> class MockP(object):
>
> # "^" is xor
>
> @icontract.pre(lambda symbol, astnode: (symbol is None) ^ (astnode is
> None))
>
> def __init__(self, symbol=None, value=EmptySymbol, astnode=None,
> initsymtable=(,)):
>
> self.symtable = dict(initsymtable)
>
> if symbol:
>
> self.expr = Expr(value=Name(id=symbol, ctx=Load()))
>
> self.symtable = {symbol: value}
>
> else:
>
> self.expr = astnode
>
> self.frozen = False
>
> def __add__(self, other):
>
> wrapped = MockP.wrap_value(other)
>
> return MockP(astnode=Expr(value=BinOp(self.expr, Add(),
> wrapped.expr), initsymtable={**self.symtable, **wrapped.symtable})
>
> def compile(self): ...
>
> def freeze(self):
>
> # frozen objects wouldn’t have an overrided getattr, allowing for
> icontract to manipulate the MockP object using its public interface
>
> self.frozen = True
>
> @classmethod
>
> def wrap_value(cls, obj):
>
># create a MockP object from a value. Generate a random identifier
> and set that as the key in symtable, the AST node is the name of that
> identifier, retrieving its value through simple expression evaluation.
>
>...
>
>
> thunk = MockP.wrap_value
>
> P = MockP('P')
>
> # elsewhere: ensure P is only accessed via valid “dot attribute access”
> inside @snapshot so contracts fail early, or don’t and allow Magic like
> __dict__ to occur on P.
>
> On Sep 27, 2018, at 9:49 PM, Marko Ristin-Kaufmann 
> wrote:
>
> Hi James,
>
> I still have a feeling that most developers would like to store the state
> in many different custom ways. I see also thunk and snapshot with wrapper
> objects to be much more complicated to implement and maintain; I'm thinking
> mostly about all the edge cases which we would not be able to cover (and
> how complex that would be to cover them). Then the linters need also to
> work around such wrappers... It might also scare users off since it looks
> like too much magic. Another concern I also have is that it's probably very
> hard to integrate these wrappers with mypy later -- but I don't really have
> a clue about that, only my gut feeling?
>
> What about we accepted to repeat "lambda P, " prefix, and have something
> like this:
>
> @snapshot(
>   lambda P, some_name: len(P.some_property),
>   lambda P, another_name: hash(P.another_property)
> )
>
> It's not too verbose for me and you can still explain in three-four
> sentences what happens below the hub in the library's docs.  A
> pycharm/pydev/vim/emacs plugins could hide the verbose parts.
>
> I performed a small experiment to test how this solution plays with pylint
> and it seems OK that arguments are not used in lambdas.
>
> Cheers,
> Marko
>
>
> On Thu, 27 Sep 2018 at 12

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi,

(Posting from work, so sorry for the short response.)

@Paul Moore  icontract.pre/post/inv have the enabled
argument; if not enabled, the contract is ignored.

Similarly with rmdir() -- "the directory must be empty" -- but how exactly
>> am I supposed to check that?
>>
>
> Isn't that the whole point? The prose statement "the directory must be
> empty" is clear. But the exact code check isn't - and may be best handled
> by a series of unit tests, rather than a precondition.
>

I meant "check" as a user, not as a developer. As in "What did the
implementer think -- how am I supposed to check that the directory is
empty?" A la: "Dear user, if you want to rmdir, here is what you need to
check that it is indeed a dir, and here is what you need to check that it
is empty. If both checks pass, run me."

@David patching __doc__ automatically is on the short-term todo list. I
suppose we'll just add sphinx directives (:requires:, :ensures: etc.)

* Marko isn't that familiar with the codebase, so there may be better
> ways to express certain things
>

This is true :)

* Sometimes it's just plain hard to express a verbal constraint in code
>

In these cases you simply don't express it in code. Why would you? If it's
complex code, possibility that you have an error is probably equal or
higher than that your comment rots.

@pre(lambda args, result: not any(Path(arg).is_absolute() for arg in args)
> or
> (result == [pth for arg in args for pth in [Path(arg)] if
> pth.is_absolute()][-1]),
> "When several absolute paths are given, the last is taken as an anchor
> (mimicking os.path.join()’s behaviour)")
>

I'm really not familiar with the code base nor with how to write stuff nice
and succinct in python. This particular contract was hard to define because
there were no last() and no arg_is_absolute() functions.

Otherwise, it would have read:

@pre(lambda args, result: not any(arg_is_absolute(arg) for arg in args) or
result == Path(last(arg for arg in args if arg_is_absolute(arg)))

When rendered, this wouldn't look too bad to read.

@Chris

> It is still fundamentally difficult to make assertions about the file
> system as pre/post contracts. Are you becoming aware of this?
> Contracts, as has been stated multiple times, look great for
> mathematically pure functions that have no state outside of their own
> parameters and return values (and 'self', where applicable), but are
> just poor versions of unit tests when there's anything external to
> consider.
>

I never thought of these particular contracts as running in the production.
I would set them to run only in testing and only on part of the tests where
they are safe from race conditions (e.g., setting
enabled=test_pathlib.SERIAL; toggling mechanism is something I haven't
spent too much thought about either and would also need to be
discussed/implemented.).

I really thought about them as documentation, not for correctness (or at
best deeper tests during the testing in a "safe" local environment, for
example when you want to check if all the contracts also hold on situations
in *my *testsuite, not only during the test suite of pathlib).

In the end, I'm calling it the day. I really got tired in the last few
days. Standardizing contracts for python is not worth the pain. We'll
continue to develop icontract for our internal needs and keep it open
source, so anybody who's interested can have a look. Thank you all for a
very lively discussions!

Cheers,
Marko



On Fri, 28 Sep 2018 at 14:49, Paul Moore  wrote:

> On Fri, 28 Sep 2018 at 13:23, David Mertz  wrote:
> > I agree that all the Sphinx documentation examples shown are very nice.
> Prose descriptions would often be nicer still, but the Boolean expressions
> are helpful for those unusual cases where I don't want to look at the code.
>
> I'm ambivalent about the Sphinx examples. I find the highly detailed
> code needed to express a condition fairly unreadable (and I'm an
> experienced Python programmer). For example
>
> @pre(lambda args, result: not any(Path(arg).is_absolute() for arg in args)
> or
> (result == [pth for arg in args for pth in [Path(arg)] if
> pth.is_absolute()][-1]),
> "When several absolute paths are given, the last is taken as an anchor
> (mimicking os.path.join()’s behaviour)")
>
> The only way I'd read that is by looking at the summary text - I'd
> never get the sense of what was going on from the code alone. There's
> clearly a number of trade-offs going on here:
>
> * Conditions should be short, to avoid clutter
> * Writing helper functions that are *only* used in conditions is more
> code to test or get wrong
> * Sometimes it's just plain hard to express a verbal constraint in code
> * Marko isn't that familiar with the codebase, so there may be better
> ways to express certain things
>
> But given that *all* the examples I've seen of contracts have this
> issue (difficult to read expressions) I suspect the problem is
> inherent.
>
> Another thing that I haven't yet seen clearly explained. 

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

I supposed to
know what "like" means?

Similarly with rmdir() -- "the directory must be empty" -- but how exactly
am I supposed to check that?

Anyhow, please have a look at the contracts and let me know what you think.
Please consider it an illustration. Try to question whether the contracts I
wrote are so obvious to everybody even if they are obvious to you and keep
in mind that the user does not look into the implementation. And please try
to abstract away the aesthetics: neither icontract library that I wrote nor
the sphinx extension are of sufficient quality. We use them for our company
code base, but they still need a lot of polishing. So please try to focus
only on the content. We are still talking about contracts in general, not
about the concrete contract implementation.

Cheers,
Marko


On Thu, 27 Sep 2018 at 11:37, Marko Ristin-Kaufmann 
wrote:

> Hi Paul,
> I only had a contracts library in mind (standardized so that different
> modules with contracts can interact and that the ecosystem for automic
> testing could emerge). I was never thinking about the philosophy or design
> methodology (where you write _all_ the contracts first and then have the
> implementation fulfill them). I should have clarified that more. I
> personally also don't think that such a methodology is practical.
>
> I really see contracts as verifiable docs that rot less fast than human
> text and are formally precise / less unambiguous than human text. Other
> aspects such as deeper tests and hand-braking (e.g., as postconditions
> which can't be practically implemented in python without exit stack context
> manager) are also nice to have.
>
> I should be done with pathlib contracts by tonight if I manage to find
> some spare time in the evening.
>
> Cheers,
> Marko
>
> Le jeu. 27 sept. 2018 à 10:43, Paul Moore  a écrit :
>
>> On Thu, 27 Sep 2018 at 08:03, Greg Ewing 
>> wrote:
>> >
>> > David Mertz wrote:
>> > > the reality is that they really ARE NOT much different
>> > > from assertions, in either practice or theory.
>> >
>> > Seems to me that assertions is exactly what they are. Eiffel just
>> > provides some syntactic sugar for dealing with inheritance, etc.
>> > You can get the same effect in present-day Python if you're
>> > willing to write the appropriate code.
>>
>> Assertions, as far as I can see, are the underlying low level
>> *mechanism* that contracts would use. Just like they are the low level
>> mechanism behind unit tests (yeah, it's really exceptions, but close
>> enough). But like unit tests, contracts seem to me to be a philosophy
>> and a library / programming technique layered on top of that base. The
>> problem seems to me to be that DbC proponents tend to evangelise the
>> philosophy, and ignore requests to show the implementation (often
>> pointing to Eiffel as an "example" rather than offering something
>> appropriate to the language at hand). IMO, people don't tend to
>> emphasise the "D" in DbC enough - it's a *design* approach, and more
>> useful in that context than as a programming construct.
>>
>> For me, the philosophy seems like a reasonable way of thinking, but
>> pretty old hat (I learned about invariants and pre-/post-conditions
>> and their advantages for design when coding in PL/1 in the 1980's,
>> about the same time as I was learning Jackson Structured Programming).
>> I don't think in terms of contracts as often as I should - but it's
>> unit tests that make me remember to do so. Would a dedicated
>> "contracts" library help? Probably not much, but maybe (if it were
>> lightweight enough) I could get used to the idea.
>>
>> Like David, I find that having contracts inline is the biggest problem
>> with them. I try to keep my function definitions short, and contracts
>> can easily add 100% overhead in terms of lines of code. I'd much
>> prefer contracts to be in a separate file. (Which is basically what
>> unit tests written with DbC as a principle in mind would be). If I
>> have a function definition that's long enough to benefit from
>> contracts, I'd usually think "I should refactor this" rather than "I
>> should add contracts".
>>
>> Paul
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Paul,
I only had a contracts library in mind (standardized so that different
modules with contracts can interact and that the ecosystem for automic
testing could emerge). I was never thinking about the philosophy or design
methodology (where you write _all_ the contracts first and then have the
implementation fulfill them). I should have clarified that more. I
personally also don't think that such a methodology is practical.

I really see contracts as verifiable docs that rot less fast than human
text and are formally precise / less unambiguous than human text. Other
aspects such as deeper tests and hand-braking (e.g., as postconditions
which can't be practically implemented in python without exit stack context
manager) are also nice to have.

I should be done with pathlib contracts by tonight if I manage to find some
spare time in the evening.

Cheers,
Marko

Le jeu. 27 sept. 2018 à 10:43, Paul Moore  a écrit :

> On Thu, 27 Sep 2018 at 08:03, Greg Ewing 
> wrote:
> >
> > David Mertz wrote:
> > > the reality is that they really ARE NOT much different
> > > from assertions, in either practice or theory.
> >
> > Seems to me that assertions is exactly what they are. Eiffel just
> > provides some syntactic sugar for dealing with inheritance, etc.
> > You can get the same effect in present-day Python if you're
> > willing to write the appropriate code.
>
> Assertions, as far as I can see, are the underlying low level
> *mechanism* that contracts would use. Just like they are the low level
> mechanism behind unit tests (yeah, it's really exceptions, but close
> enough). But like unit tests, contracts seem to me to be a philosophy
> and a library / programming technique layered on top of that base. The
> problem seems to me to be that DbC proponents tend to evangelise the
> philosophy, and ignore requests to show the implementation (often
> pointing to Eiffel as an "example" rather than offering something
> appropriate to the language at hand). IMO, people don't tend to
> emphasise the "D" in DbC enough - it's a *design* approach, and more
> useful in that context than as a programming construct.
>
> For me, the philosophy seems like a reasonable way of thinking, but
> pretty old hat (I learned about invariants and pre-/post-conditions
> and their advantages for design when coding in PL/1 in the 1980's,
> about the same time as I was learning Jackson Structured Programming).
> I don't think in terms of contracts as often as I should - but it's
> unit tests that make me remember to do so. Would a dedicated
> "contracts" library help? Probably not much, but maybe (if it were
> lightweight enough) I could get used to the idea.
>
> Like David, I find that having contracts inline is the biggest problem
> with them. I try to keep my function definitions short, and contracts
> can easily add 100% overhead in terms of lines of code. I'd much
> prefer contracts to be in a separate file. (Which is basically what
> unit tests written with DbC as a principle in mind would be). If I
> have a function definition that's long enough to benefit from
> contracts, I'd usually think "I should refactor this" rather than "I
> should add contracts".
>
> Paul
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi David,
I'm writing contracts for pathlib as a proof-of-concept.

See pypackagery for an example of a project on pypi that uses contracts:
https://pypi.org/project/pypackagery/

The docs with contracts are available at:
https://pypackagery.readthedocs.io/en/latest/packagery.html

Mind that the interface to icontract might change soon so I'd wait a month
or two before it stabilizes.

Cheers,
Marko

Le mer. 26 sept. 2018 à 20:12, David Stanek  a écrit :

> On Wed, Sep 26, 2018 at 12:49 AM Marko Ristin-Kaufmann
>  wrote:
> >
> >> An extraordinary claim is like "DbC can improve *every single project*
> >> on PyPI". That requires a TON of proof. Obviously we won't quibble if
> >> you can only demonstrate that 99.95% of them can be improved, but you
> >> have to at least show that the bulk of them can.
> >
> >
> > I tried to give the "proof" (not a formal one, though) in my previous
> message.
> >
>
> I have to admit that I haven't kept up with the discussion today, but
> I was also hoping to see some proof. I'm genuinely interested in
> seeing if this is something that can help me and the teams I work
> with. I was very interested in DbC a long time ago, but never found a
> way to make it valuable to me.
>
> I'd like to see a project from PyPI converted to use DbC. This would
> make it easy to see the real world difference between an
> implementation developed using DbC compared to one that is well
> documented, tested and maybe even includes type hints. Toy or
> cherry-picked examples don't help me get a feel for it.
>
> --
> david stanek
> web: https://dstanek.com
> twitter: https://twitter.com/dstanek
> linkedin: https://www.linkedin.com/in/dstanek/
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Paul,
Quite a few people replied on this thread and the previous one before the
fork that dbc is either useless in Python or at best useful in
avionics/niche applications.

I'm really only saying that contracts are a superior (complementary) tool
to informal documentation, doctests, reading the implementation, reading
the tests and trial-and-error.

For every library that have the contracts which can be written down
formally in a pragmatic way, and when the users of the library are multiple
-- then these libraries would benefit from dbc.

That's all that I'm saying and it might have come over as arrogant due to
limits of the medium. It was not my intention to sound so.

I'll have a look at pathlib then.

Cheers,
Marko



Le mer. 26 sept. 2018 à 15:15, Paul Moore  a écrit :

> On Wed, 26 Sep 2018 at 14:04, Marko Ristin-Kaufmann
>  wrote:
> >
> > @Chris Angelico  would annotating pathlib convince you that contracts
> are useful? Is it general enough to start with?
>
> Whoa - be careful here. No-one is saying that "contracts aren't
> useful" (at least not that I'd heard). We're saying that contracts
> *aren't a solution for every library in existence* (which was
> essentially your claim). Annotating pathlib can't convince anyone of
> that claim. At best (and this is all that I'm imagining) it might give
> some indication of why you have such an apparently-unreasonable level
> of confidence in contracts.
>
> I do not expect *ever* to believe you when you say that all projects
> would benefit from contracts. What I might get from such an exercise
> is a better understanding of what you're imagining when you talk about
> a real project "using contracts". Best case scenario - I might be
> persuaded to use them occasionally. Worst case scenario - I find them
> distracting and unhelpful, and we agree to differ on their value.
>
> Paul
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Paul,

Are we talking here about coding explicit executable contracts in the
> source code of the library, or using (formally described in terms of
> (pseudo-)code) contract-style descriptions in the documentation, or
> simply using the ideas of contract-based thinking in designing and
> writing code?
>

The current implementation of icontract uses decorators to decorate the
functions and classes (and metaclasses to support inheritance of
contracts). You have an "enabled" flag which you can set on/off if you want
to disable the contract in some situations.

We are talking about the explicit executable contracts :).

You didn't address my question "does this apply to the stdlib"? If it
> doesn't, your argument has a huge hole - how did you decide that the
> solution you're describing as "beneficial to all libraries" doesn't
> improve the stdlib? If it does, then why not demonstrate your case?
> Give concrete examples - look at some module in the stdlib (for
> example, pathlib) and show exactly what contracts you'd add to the
> code, what the result would look like to the library user (who
> normally doesn't read the source code) and to the core dev (who does).
> Remember that pathlib (like all of the stdlib) doesn't use type
> annotations, and that is a *deliberate* choice, mandated by Guido when
> he first introduced type annotations. So you're not allowed to add
> contracts like "arg1 is a string", nor are you allowed to say that the
> lack of type annotations makes the exercise useless.


Sorry, I missed that point; the messages are getting long :) Yes, the
contracts would make sense in stdlib as well, I'd say.

@Chris Angelico   would annotating pathlib convince you
that contracts are useful? Is it general enough to start with?

On Wed, 26 Sep 2018 at 14:58, Paul Moore  wrote:

> On Wed, 26 Sep 2018 at 13:40, Marko Ristin-Kaufmann
>  wrote:
>
> > Please mind that I said: any library would benefit from it, as in the
> users of any library on pipy would benefit from better, formal and more
> precise documentation. That doesn't imply that all the contracts need to be
> specified or that you have to specify the contract for every function, or
> that you omit the documentation altogether. Some contracts are simply too
> hard to get explicitly. Some are not meaningful even if you could write
> them down. Some you'd like to add, but run only at test time since too slow
> in production. Some run in production, but are not included in the
> documentation (e.g., to prevent the system to enter a bad state though it
> is not meaningful for the user to actually read that contract).
> >
> > Since contracts are formally written, they can be verified. Human text
> can not. Specifying all the contracts is in most cases not meaningful. In
> my day-to-day programming, I specify contracts on the fly and they help me
> express formally to the next girl/guy who will use my code what to expect
> or what not. That does not mean that s/he can just skip the documentation
> or that contracts describe fully what the function does. They merely help
> -- help him/her what arguments and results are expected. That does not mean
> that I fully specified all the predicates on the arguments and the result.
> It's merely a help à la
> > * "Ah, this argument needs to be bigger than that argument!"
> > * "Ah, the resulting dictionary is shorter than the input list!"
> > * "Ah, the result does not include the input list"
> > * "Ah, this function accepts only files (no directories) and relative
> paths!"
> > * "Ah, I put the bounding box outside of the image -- something went
> wrong!"
> > * "Ah, this method allows me to put the bounding box outside of the
> image and will fill all the outside pixels with black!" etc.
>
> Whoops, I think the rules changed under me again :-(
>
> Are we talking here about coding explicit executable contracts in the
> source code of the library, or using (formally described in terms of
> (pseudo-)code) contract-style descriptions in the documentation, or
> simply using the ideas of contract-based thinking in designing and
> writing code?
>
> > For example, if I have an object detector operating on a
> region-of-interest and returning bounding boxes of the objects, the
> postconditions will not be: "all the bounding boxes are cars", that would
> impossible. But the postcondition might check that all the bounding boxes
> are within the region-of-interest or slightly outside, but not completely
> outside etc.
>
> I understand that you have to pick an appropriate level of strictness
> when writing contracts. That's not ever been in question (at least in
> my mind

Re: [Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

Hi James,

Actually, following on #A4, you could also write those as multiple
decorators:
@snpashot(lambda _, some_identifier: some_func(_, some_argument.some_attr)
@snpashot(lambda _, other_identifier: other_func(_.self))

Am I correct?

"_" looks a bit hard to read for me (implying ignored arguments).

Why uppercase "P" and not lowercase (uppercase implies a constant for me)?
Then "O" for "old" and "P" for parameters in a condition:
@post(lambda O, P: ...)
?

It also has the nice property that it follows both the temporal and the
alphabet order :)

On Wed, 26 Sep 2018 at 14:30, James Lu  wrote:

> I still prefer snapshot, though capture is a good name too. We could use
> generator syntax and inspect the argument names.
>
> Instead of “a”, perhaps use “_”. Or maybe use “A.”, for arguments. Some
> people might prefer “P” for parameters, since parameters sometimes means
> the value received while the argument means the value passed.
>
> (#A1)
>
> from icontract import snapshot, __
> @snapshot(some_func(_.some_argument.some_attr) for some_identifier, _ in
> __)
>
> Or (#A2)
>
> @snapshot(some_func(some_argument.some_attr) for some_identifier, _,
> some_argument in __)
>
> —
> Or (#A3)
>
> @snapshot(lambda some_argument,_,some_identifier:
> some_func(some_argument.some_attr))
>
> Or (#A4)
>
> @snapshot(lambda _,some_identifier: some_func(_.some_argument.some_attr))
> @snapshot(lambda _,some_identifier, other_identifier:
> some_func(_.some_argument.some_attr), other_func(_.self))
>
> I like #A4 the most because it’s fairly DRY and avoids the extra
> punctuation of
>
> @capture(lambda a: {"some_identifier": some_func(a.some_argument.some_attr)})
>
>
> On Sep 26, 2018, at 12:23 AM, Marko Ristin-Kaufmann <
> marko.ris...@gmail.com> wrote:
>
> Hi,
>
> Franklin wrote:
>
>> The name "before" is a confusing name. It's not just something that
>> happens before. It's really a pre-`let`, adding names to the scope of
>> things after it, but with values taken before the function call. Based
>> on that description, other possible names are `prelet`, `letbefore`,
>> `predef`, `defpre`, `beforescope`. Better a name that is clearly
>> confusing than one that is obvious but misleading.
>
>
> James wrote:
>
>> I suggest that instead of “@before” it’s “@snapshot” and instead of “old”
>> it’s “snapshot”.
>
>
> I like "snapshot", it's a bit clearer than prefixing/postfixing verbs with
> "pre" which might be misread (*e.g., *"prelet" has a meaning in Slavic
> languages and could be subconsciously misread, "predef" implies to me a pre-
> *definition* rather than prior-to-definition , "beforescope" is very
> clear for me, but it might be confusing for others as to what it actually
> refers to ). What about "@capture" (7 letters for captures *versus *8 for
> snapshot)? I suppose "@let" would be playing with fire if Python with
> conflicting new keywords since I assume "let" to be one of the candidates.
>
> Actually, I think there is probably no way around a decorator that
> captures/snapshots the data before the function call with a lambda (or even
> a separate function). "Old" construct, if we are to parse it somehow from
> the condition function, would limit us only to shallow copies (and be
> complex to implement as soon as we are capturing out-of-argument values
> such as globals *etc.)*. Moreove, what if we don't need shallow copies? I
> could imagine a dozen of cases where shallow copy is not what the
> programmer wants: for example, s/he might need to make deep copies, hash or
> otherwise transform the input data to hold only part of it instead of
> copying (*e.g., *so as to allow equality check without a double copy of
> the data, or capture only the value of certain property transformed in some
> way).
>
> I'd still go with the dictionary to allow for this extra freedom. We could
> have a convention: "a" denotes to the current arguments, and "b" denotes
> the captured values. It might make an interesting hint that we put "b"
> before "a" in the condition. You could also interpret "b" as "before" and
> "a" as "after", but also "a" as "arguments".
>
> @capture(lambda a: {"some_identifier": some_func(a.some_argument.some_attr)})
> @post(lambda b, a, result: b.some_identifier > result + 
> a.another_argument.another_attr)
> def some_func(some_argument: SomeClass, another_argument: AnotherClass) -> 
> SomeResult:
> ...
>
> "b" can be omit

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

P.S. My offer still stands: I would be very glad to annotate with contracts
a set of functions you deem representative (*e.g., *from a standard library
or from some widely used library). Then we can discuss how these contracts.
It would be an inaccurate estimate of the benefits of DbC in Python, but
it's at least better than no estimate. We can have as little as 10
functions for the start. Hopefully a couple of other people would join, so
then we can even see what the variance of contracts would look like.

On Wed, 26 Sep 2018 at 14:40, Marko Ristin-Kaufmann 
wrote:

> Hi Chris and Paul,
>
> Let me please answer your messages in one go as they are related.
>
> Paul wrote:
>
>> For most single use (or infrequently used) functions, I'd argue that the
>> trade-off *isn't* worth it.
>>
>> Here's a quick example from the pip codebase:
>>
>> # Retry every half second for up to 3 seconds
>> @retry(stop_max_delay=3000, wait_fixed=500)
>> def rmtree(dir, ignore_errors=False):
>> shutil.rmtree(dir, ignore_errors=ignore_errors,
>>   onerror=rmtree_errorhandler)
>
>
> Absolutely, I agree. If it's a single-use or infrequently used function
> that hardly anybody uses, it's not worth it. Moreover, if some contracts
> are harder to figure out than the implementation, then they are probably in
> most cases not worth the effort, too.
>
> Please mind that I said: any *library* would benefit from it, as in the
> users of any library on pipy would benefit from better, formal and more
> precise documentation. That doesn't imply that all the contracts need to be
> specified or that you have to specify the contract for *every *function,
> or that you omit the documentation altogether. Some contracts are simply
> too hard to get explicitly. Some are not meaningful even if you could write
> them down. Some you'd like to add, but run only at test time since too slow
> in production. Some run in production, but are not included in the
> documentation (*e.g., *to prevent the system to enter a bad state though
> it is not meaningful for the user to actually *read* that contract).
>
> Since contracts are formally written, they can be verified. Human text *can
> not*. Specifying all the contracts is in most cases *not *meaningful. In
> my day-to-day programming, I specify contracts on the fly and they help me
> express formally to the next girl/guy who will use my code what to expect
> or what not. That does not mean that s/he can just skip the documentation
> or that contracts describe fully what the function does. They merely help
> -- help him/her what arguments and results are expected. That *does not
> mean *that I fully specified all the predicates on the arguments and the
> result. It's merely a help à la
> * "Ah, this argument needs to be bigger than that argument!"
> * "Ah, the resulting dictionary is shorter than the input list!"
> * "Ah, the result does not include the input list"
> * "Ah, this function accepts only files (no directories) and relative
> paths!"
> * "Ah, I put the bounding box outside of the image -- something went
> wrong!"
> * "Ah, this method allows me to put the bounding box outside of the image
> and will fill all the outside pixels with black!"
>
> *etc.*
> For example, if I have an object detector operating on a
> region-of-interest and returning bounding boxes of the objects, the
> postconditions will not be: "all the bounding boxes are cars", that would
> impossible. But the postcondition might check that all the bounding boxes
> are within the region-of-interest or slightly outside, but not completely
> outside *etc.*
>
> Let's be careful not to make a straw-man here, *i.e. *to push DbC *ad
> absurdum *and then discard it that way.
>
> Also, the implicit contracts code currently has are typically pretty
>> loose. What you need to "figure out" is very general. Explicit contracts
>> are typically demonstrated as being relatively strict, and figuring out and
>> writing such contracts is more work than writing code with loose implicit
>> contracts. Whether the trade-off of defining tight explicit contracts once
>> vs inferring a loose implicit contract every time you call the function is
>> worth it, depends on how often the function is called. For most single use
>> (or  infrequently used)  functions, I'd argue that the trade-off *isn't*
>> worth it.
>
>
> I don't believe such an approach would ever be pragmatical, *i.e. *automatic
> versioning based on the contracts. It might hint it (as a warning: you
> changed a contract, but did not bump the version), but relying solely on
> this mechanism to get the versioning woul

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

t there is a list of figures
> and that they have a property "displayed" and that "State.blocked" global
> variable refers to whether the interface is blocked or not::
> > @post(lambda: all(figure.displayed for figure in figures)
> > @post(lambda: not ipython.in_pylab_mode() or not State.blocked)
> > @post(lambda: not interactive() or State.blocked)
> > matplotlib.pyplot.show()
> >
>
> There is no such thing as "State.blocked". It blocks. The function
> *does not return* until the figure has been displayed, and dismissed.
> There's no way to recognize that inside the function's state.
>
> Contracts are great when every function is 100% deterministic and can
> maintain invariants and/or transform from one set of invariants to
> another. Contracts are far less clear when the definitions are
> muddier.
>

Sorry, it has been ages that I used matplotlib. I thought it meant
"blocking" as in "the drawing thread blocks". Since blocking basically
means halting the execution-- then the contracts can't help. They are
checked *before *and *after *the function executes. They can not solve the
halting problem. For that you need formal methods (and I doubt that formal
methods would ever work for matplotlib). The contracts *do not check *what
happens *during *the execution of the function. They are not meant to do
that. Even the invariants of the class are checked *before *and *after the
call to public methods *(and the private methods are allowed to break
them!).

Please mind that this is actually not the argument pro/against the
contracts -- you are discussing in this particular case a tool (different
to DbC) which should test the behavior *during the execution *of a function.

Chirs wrote:

> > However, contracts can be useful when testing the GUI -- often it is
> difficult to script the user behavior. What many people do is record a
> session and re-play it. If there is a bug, fix it. Then re-record. While
> writing unit tests for GUI is hard since GUI changes rapidly during
> development and scripting formally the user behavior is tedious, DbC might
> be an alternative where you specify as much as you can, and then just
> re-run through the session. This implies, of course, a human tester.
> >
>
> That doesn't sound like the function's contract. That sounds like a
> test case - of which you would have multiple, using different
> "scripted session" inputs and different outputs (some of success, some
> of expected failure).
>

Well, yes, you can view it as a testing technique; it assumes that
scripting a session is often difficult for GUIs and sometimes harder (since
combinatorial) than the implementation itself. What I saw people do is
write the contracts, put the program in debug mode and let the human tester
test it. Think of it as a random test where only checks are your
pre/postconditions. The human annotator mimics a meaningful random walk and
uses the application as s/he sees fit. I'm not championing this approach,
just noting it here as a potential use case.

There's certainly benefits for the "contracts as additional tests"
> viewpoint. But whenever that's proposed as what people understand by
> DbC, the response is "no, it's not like that at all". So going back to
> the "why isn't DbC more popular" question - because no-one can get a
> clear handle on whether they are "like tests" or "like assertions" or
> "something else" :-)


I think that it's a tool that you can use for many things:
* verifiable documentation
* deeper testing (every test case tests also other parts of the system,
akin to asserts)
* automatic test generation
* hand-break akin to asserts

I find the first one, verifiable documentation, to be the most useful one
in working with my team at the moment.

Cheers,
Marko

On Wed, 26 Sep 2018 at 10:59, Paul Moore  wrote:

> On Wed, 26 Sep 2018 at 09:45, Chris Angelico  wrote:
> >
> > On Wed, Sep 26, 2018 at 6:36 PM Paul Moore  wrote:
> > >
> > > On Wed, 26 Sep 2018 at 06:41, Chris Angelico  wrote:
> > > >
> > > > On Wed, Sep 26, 2018 at 2:47 PM Marko Ristin-Kaufmann
> > > >  wrote:
> > > > > * The contracts written in documentation as human text inevitably
> rot and they are much harder to maintain than automatically verified formal
> contracts.
> > > >
> > > > Agreed.
> > >
> > > Agreed, if contracts are automatically verified. But when runtime cost
> > > comes up, people suggest that contracts can be disabled in production
> > > code - which invalidates the "automatically verified" premise.
> >
> > Even if they're only verified as a dedicated testing pass ("okay,
> > let's r

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

;unless ...to interactive mode": this would be actually really more
readable as a formal contract. It would imply some refactoring to add a
property when a figure was created (before or after the change to
interactive mode from non-interactive mode). Right now, it's not very clear
how you can test that even yourself as a caller. And what does "(not
recommended)" mean? Prohibited?

The blocking behavior and the corresponding argument is hard to parse from
the description for me. Writing it as a contract would, IMO, be much more
readable.

Here is my try at the contracts. Assuming that there is a list of figures
and that they have a property "displayed" and that "State.blocked" global
variable refers to whether the interface is blocked or not::
@post(lambda: all(figure.displayed for figure in figures)
@post(lambda: not ipython.in_pylab_mode() or not State.blocked)
@post(lambda: not interactive() or State.blocked)
matplotlib.pyplot.show()

The function that actually displays the figure could ensure that the
"displayed" property of the figure is set and that the window associated
with the figure is visible.

Cheers,
Marko


On Wed, 26 Sep 2018 at 09:19, Chris Angelico  wrote:

> On Wed, Sep 26, 2018 at 5:10 PM Marko Ristin-Kaufmann
>  wrote:
> > The original objection was that DbC in general is not beneficial; not
> that there are lacking tools for it (this probably got lost in the many
> messages on this thread). If you assume that there are no suitable tools
> for DbC, then yes, DbC is certainly not beneficial to any project since
> using it will be clumsy and difficult. It's a chicken-and-egg problem, so
> we need to assume that there are good tools for DbC in order for it to be
> beneficial.
> >
> >> Disagreed. I would most certainly NOT assume that every reader knows
> >> any particular syntax for such contracts. However, this is a weaker
> >> point.
> >
> >
> > The contracts are written as boolean expressions. While I agree that
> many programmers have a hard time with boolean expressions and quantifiers,
> I don't see this as a blocker to DbC. There is no other special syntax for
> DbC unless we decide to add it to the core language (which I don't see as
> probable). What I would like to have is a standard library so that
> inter-library interactions based on contracts are possible and an ecosystem
> could emerge around it.
> >
>
> It's easy to say that they're boolean expressions. But that's like
> saying that unit tests are just a bunch of boolean expressions too.
> Why do we have lots of different forms of test, rather than just a big
> fat "assert this and this and this and this and this and this"?
> Because the key to unit testing is not "boolean expressions", it's a
> language that can usefully describe what it is we're testing.
> Contracts aren't just boolean expressions - they're a language (or a
> mini-language) that lets you define WHAT the contract entails.
>
> >> You might argue that a large proportion of PyPI projects will be
> >> "library-style" packages, where the main purpose is to export a bunch
> >> of functions. But even then, I'm not certain that they'd all benefit
> >> from DbC.
> >
> >
> > Thanks for this clarification (and the download-yt example)! I actually
> only had packages-as-libraries in mind, not the executable scripts; my
> mistake. So, yes, "any Pypi package" should be reformulated to "any library
> on Pypi" (meant to be used by a wider audience than the developers
> themselves).
> >
>
> Okay. Even with that qualification, though, I still think that not
> every library will benefit from this. For example, matplotlib's
> plt.show() method guarantees that... a plot will be shown, and the
> user will have dismissed it, before it returns. Unless you're inside
> Jupyter/iPython, in which case it's different. Or if you're in certain
> other environments, in which case it's different again. How do you
> define the contract for something that is fundamentally interactive?
>
> You can define very weak contracts. For instance, input() guarantees
> that its return value is a string. Great! DbC doing the job of type
> annotations. Can you guarantee anything else about that string? Is
> there anything else useful that can be spelled easily?
>
> > I totally agree. The discussion related to DbC in my mind always
> revolved around these use cases where type annotations are beneficial as
> well. Thanks for pointing that out and I'd like to apologize for the
> confusion! For the future discussion, let's focus on these use cases and
> please do ignore the rest. I'd still say that there is a plethora of
> libraries publish

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Chris,

> * There are always contracts, they can be either implicit or explicit.
> You need always to figure them out before you call a function or use its
> result.
>
> Not all code has such contracts. You could argue that code which does
> not is inferior to code which does, but not everything follows a
> strictly-definable pattern.
>

Well, you need to know what to supply to a function and what to expect, at
least subconsciously. The question is just whether you can formulate these
contracts or not; and whether you do or not.

> * The are tools for formal contracts.
>
> That's the exact point you're trying to make, so it isn't evidence for
> itself. Tools for formal contracts exist as third party in Python, and
> if that were good enough for you, we wouldn't be discussing this.
> There are no such tools in the standard library or language that make
> formal contracts easy.
>

The original objection was that DbC in general is not beneficial; not that
there are lacking tools for it (this probably got lost in the many messages
on this thread). If you assume that there are no suitable tools for DbC,
then yes, DbC is certainly *not *beneficial to any project since using it
will be clumsy and difficult. It's a chicken-and-egg problem, so we need to
assume that there are good tools for DbC in order for it to be beneficial.

Disagreed. I would most certainly NOT assume that every reader knows
> any particular syntax for such contracts. However, this is a weaker
> point.


The contracts are written as boolean expressions. While I agree that many
programmers have a hard time with boolean expressions and quantifiers, I
don't see this as a blocker to DbC. There is no other special syntax for
DbC unless we decide to add it to the core language (which I don't see as
probable). What I would like to have is a standard library so that
inter-library interactions based on contracts are possible and an ecosystem
could emerge around it.

Quite some people object that DbC should be rejected as their "average
programmer Joe/Jane" can't deal with the formal expressions. Some readers
have problems parsing "all(i > 0 for i in result) or len(result) < 3" and
can only parse "All numbers in the result are positive and/or the result
has fewer than 3 items". There are also programmers who have a hard time
reading "all(key == value.some_property for key, value in result.items())"
and could only read "The resulting values in the dictionary are keyed by
their some_property attribute.".

I hope that education in computer science is improving and that soon
programmers will be able to read these formal expressions. I'm also not
sure if that is the case for non-English speakers. Hence I assumed that the
readers of the contracts are familiar with formal boolean expressions. If
you assume that readers have hard time parsing quantifiers and boolean
logic then DbC is again certainly not beneficial.

It would make me sad if a feature is rejected on grounds that we have to
accept that many programmers don't master the basics of computer science
(which I consider the boolean expressions to be).

You might argue that a large proportion of PyPI projects will be
> "library-style" packages, where the main purpose is to export a bunch
> of functions. But even then, I'm not certain that they'd all benefit
> from DbC.
>

Thanks for this clarification (and the download-yt example)! I actually
only had packages-as-libraries in mind, not the executable scripts; my
mistake. So, yes, "any Pypi package" should be reformulated to "any library
on Pypi" (meant to be used by a wider audience than the developers
themselves).

People have said the same thing about type checking, too. Would
> *every* project on PyPI benefit from MyPy's type checks? No. Syntax
> for them was added, not because EVERYONE should use them, but  because
> SOME will use them, and it's worth having some language support. You
> would probably do better to argue along those lines than to try to
> claim that every single project ought to be using contracts.


I totally agree. The discussion related to DbC in my mind always revolved
around these use cases where type annotations are beneficial as well.
Thanks for pointing that out and I'd like to apologize for the confusion!
For the future discussion, let's focus on these use cases and please do
ignore the rest. I'd still say that there is a plethora of libraries
published on Pypi (Is there a way to find out the stats?).

but I'm still -0.5 on adding anything of the sort to the stdlib, as I
> don't yet see that *enough* projects would actually benefit.
>

Please see my previous message -- could you maybe say what would convince
you that enough projects would actually benefit from formal contracts?

Cheers,
Marko

On Wed, 26 Sep 2018 at 07:40, Chris Angelico  wrote:

>

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Chris,

It's easy to show beautiful examples that may actually depend on other
> things. Whether that's representative of all contracts is another
> question.


I agree. There are also many contracts which are simply too hard to write
down formally. But many are also easily captured in formal manner in my
experience. The question is, of course, how many and you make a fair point
there.

@Chris and others requesting data: my time is way too limited to provide a
large-scale code analysis of many pypi packages (family obligations with a
toddler, 9-6 job). I'm not doing research, and such a study would require
substantial time resources. Is there an alternative request that you think
that I (and other volunteers?)  could accomplish in a reasonable (free)
time? Maybe you could compile a list of 100-200 (or even less) functions
from representative modules and I try to annotate them with contracts and
we see if that's convincing? It's up to you to pick representative
functions and up to me to annotate them with contracts. That would diffuse
the argument that I intentionally picked the functions whose contracts are
easily and nice to annotate.

Cheers,
Marko

On Wed, 26 Sep 2018 at 01:20, Chris Angelico  wrote:

> On Wed, Sep 26, 2018 at 7:59 AM Kyle Lahnakoski 
> wrote:
> > I use DbC occasionally to clarify my thoughts during a refactoring, and
> then only in the places that continue to make mistakes. In general, I am
> not in a domain that benefits from DbC.
> >
> > Contracts are code: More code means more bugs.
>
> Contracts are executable documentation. If you can lift them directly
> into user-readable documentation (and by "user" here I mean the user
> of a library), they can save you the work of keeping your
> documentation accurate.
>
> > This contract does not help me:
> >
> > What is_absolute()?  is "file:///" absolute?
>
> I'd have to assume that is_absolute() is defined elsewhere. Which
> means that the value of this contract depends entirely on having other
> functions, probably ALSO contractually-defined, to explain it.
>
> > How does this code fail?
> > What does a permission access problem look like?
>
> Probably an exception. This is Python code, and I would generally
> assume that problems are reported as exceptions.
>
> > Can initial_paths can be None?
>
> This can be answered from the type declaration. It doesn't say
> Optional, so no, it can't be None.
>
> > Can initial_paths be files? directories?
>
> Presumably not a question you'd get if you were actually using it; the
> point of the function is to "[r]esolve the initial paths of the
> dependency graph by recursively adding *.py files beneath given
> directories", so you'd call it because you have directories and want
> files back.
>
> > What are the side effects?
>
> Hopefully none, other than the normal implications of hitting the file
> system.
>
> It's easy to show beautiful examples that may actually depend on other
> things. Whether that's representative of all contracts is another
> question.
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Kyle,

6) The name of the method
> 7) How the method is called throughout the codebase
>
10) relying on convention inside, and outside, the application
>

Sorry, by formulating 2) as "docstring" I excluded names of the methods as
well as variables. Please assume that 2) actually entails those as well.
They are human text and hence not automatically verifiable, hence qualify
as 2).

8) observing input and output values during debugging
> 9) observing input and output values in production
>

Sorry, again I implicitly subsumed 8-9 under 4), reading the implementation
code (including the trial-and-error). My assumption was that it is
incomparably more costly to apply trial-and-error than read the contracts
given that contracts can be formulated. Of course, not all contracts can be
formulated all the time.

11) Don't communicate - Sometimes / is too high;
> code is not repaired, only replaced.
>

I don't see this as an option for any publicly available, high-quality
module on pypi or in any organization. As I already noted in my message to
Hugh, the argument in favor of* undocumented and/or untested code* are not
the arguments. I assume we want a *maintainable* and *usable* modules. I've
never talked about undocumented throw-away exploratory code. Most of the
Python features become futile in that case (type annotations and static
type checking with mypy, to name only the few).

Does it work on Windows?
>
This is probably impossible to write as a contract, but needs to be tested
(though maybe there is a way to check it and encapsulate the check in a
separate function and put it into the contract).

What is_absolute()?  is "file:///" absolute?
>
Since the type is pathlib.Path (as written in the type annotation), it's
pathlib.Path.is_absolute() method. Please see
https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.is_absolute

At a high level, I can see the allure of DbC:  Programming can be a craft,
> and a person can derive deep personal satisfaction from perfecting the code
> they work on. DbC provides you with more decoration, more elaboration, more
> ornamentation, more control.  This is not bad, but I see all your arguments
> as personal ascetic sense.  DbC is only appealing under certain accounting
> rules.  Please consider the possibility that "the best code" is: low $$$,
> buggy, full of tangles, and mostly gets the job done.   :)

Actually, this goes totally contrary to most of my experience. Bad code is
unmaintainable and ends up being much more costly down the line. It's also
what we were taught in software engineering lectures in the university
(some 10-15 years ago) and I always assumed that the studies presented
there were correct.

Saying that writing down contracts is costly is a straw-man. It is costly
if you need to examine the function and write them down. If you *are
writing *the function and just keep adding the contracts as-you-go, it's
basically very little overhead cost. You make an assumption of the input,
and instead of just coding on, you scroll up, write it down formally, and
go back where you stopped and continue the implementation. Or you think for
a minute what contracts your function needs to expect/satisfy before you
start writing it (or during the design). I don't see how this can be less
efficient than trial-and-error and making possibly wrong assumptions based
on the output that you see without any documentation by running the code of
the module.

Cheers,
Marko

On Tue, 25 Sep 2018 at 23:59, Kyle Lahnakoski 
wrote:

>
> I use DbC occasionally to clarify my thoughts during a refactoring, and
> then only in the places that continue to make mistakes. In general, I am
> not in a domain that benefits from DbC.
>
> Contracts are code: More code means more bugs. Declarative contracts are
> succinct, but difficult to debug when wrong; I believe this because the
> debugger support for contracts is poor; There is no way to step through the
> logic and see the intermediate reasoning in complex contracts.  A contract
> is an incomplete duplication of what the code already does: at some level
> of complexity I prefer to use a duplicate independent implementation and
> compare inputs/outputs.
> Writing contracts cost time and money; and that cost should be weighed
> against the number and flexibility of the customers that use the code.  A
> one-time script, a webapp for you team, an Android app for your startup,
> fraud software, and Facebook make different accounting decisions.  I
> contend most code projects can not justify DbC.
>
>
> On 2018-09-24 03:46, Marko Ristin-Kaufmann wrote:
>
> When you are documenting a method you have the following options:
> 1) Write preconditions and postconditions formally and include them
> automatically in the documentation (*e.g., *by using icontract library).
> 2) Write precondtions and 

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Chris,

An extraordinary claim is like "DbC can improve *every single project*
> on PyPI". That requires a TON of proof. Obviously we won't quibble if
> you can only demonstrate that 99.95% of them can be improved, but you
> have to at least show that the bulk of them can.


I tried to give the "proof" (not a formal one, though) in my previous
message. The assumptions are that:
* There are always contracts, they can be either implicit or explicit. You
need always to figure them out before you call a function or use its result.
* Figuring out contracts by trial-and-error and reading the code (the
implementation or the test code) is time consuming and hard.
* The are tools for formal contracts.
* The contracts written in documentation as human text inevitably rot and
they are much harder to maintain than automatically verified formal
contracts.
* The reader is familiar with formal statements, and hence reading formal
statements is faster than reading the code or trial-and-error.

I then went on to show why I think, under these assumptions, that formal
contracts are superior as a documentation tool and hence beneficial. Do you
think that any of these assumptions are wrong? Is there a hole in my
logical reasoning presented in my previous message? I would be very
grateful for any pointers!

If these assumptions hold and there is no mistake in my reasoning, wouldn't
that qualify as a proof?

Cheers,
Marko

On Tue, 25 Sep 2018 at 21:43, Chris Angelico  wrote:

> On Wed, Sep 26, 2018 at 3:19 AM Marko Ristin-Kaufmann
>  wrote:
> >> Claiming that DbC annotations will improve the documentation of every
> >> single library on PyPI is an extraordinary claim, and such claims
> >> require extraordinary proof.
> >
> >
> > I don't know what you mean by "extraordinary" claim and "extraordinary"
> proof, respectively. I tried to show that DbC is a great tool and far
> superior to any other tools currently used to document contracts in a
> library, please see my message
> https://groups.google.com/d/msg/python-ideas/dmXz_7LH4GI/5A9jbpQ8CAAJ.
> Let me re-use the enumeration I used in the message and give you a short
> summary.
> >
>
> An ordinary claim is like "DbC can be used to improve code and/or
> documentation", and requires about as much evidence as you can stuff
> into a single email. Simple claim, low burden of proof.
>
> An extraordinary claim is like "DbC can improve *every single project*
> on PyPI". That requires a TON of proof. Obviously we won't quibble if
> you can only demonstrate that 99.95% of them can be improved, but you
> have to at least show that the bulk of them can.
>
> > There are 150K projects on pypi.org. Each one of them would benefit if
> annotated with the contracts.
>
> This is the extraordinary claim. To justify it, you have to show that
> virtually ANY project would benefit from contracts. So far, I haven't
> seen any such proof.
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

Hi,

Franklin wrote:

> The name "before" is a confusing name. It's not just something that
> happens before. It's really a pre-`let`, adding names to the scope of
> things after it, but with values taken before the function call. Based
> on that description, other possible names are `prelet`, `letbefore`,
> `predef`, `defpre`, `beforescope`. Better a name that is clearly
> confusing than one that is obvious but misleading.


James wrote:

> I suggest that instead of “@before” it’s “@snapshot” and instead of “old”
> it’s “snapshot”.


I like "snapshot", it's a bit clearer than prefixing/postfixing verbs with
"pre" which might be misread (*e.g., *"prelet" has a meaning in Slavic
languages and could be subconsciously misread, "predef" implies to me a pre-
*definition* rather than prior-to-definition , "beforescope" is very clear
for me, but it might be confusing for others as to what it actually refers
to ). What about "@capture" (7 letters for captures *versus *8 for
snapshot)? I suppose "@let" would be playing with fire if Python with
conflicting new keywords since I assume "let" to be one of the candidates.

Actually, I think there is probably no way around a decorator that
captures/snapshots the data before the function call with a lambda (or even
a separate function). "Old" construct, if we are to parse it somehow from
the condition function, would limit us only to shallow copies (and be
complex to implement as soon as we are capturing out-of-argument values
such as globals *etc.)*. Moreove, what if we don't need shallow copies? I
could imagine a dozen of cases where shallow copy is not what the
programmer wants: for example, s/he might need to make deep copies, hash or
otherwise transform the input data to hold only part of it instead of
copying (*e.g., *so as to allow equality check without a double copy of the
data, or capture only the value of certain property transformed in some
way).

I'd still go with the dictionary to allow for this extra freedom. We could
have a convention: "a" denotes to the current arguments, and "b" denotes
the captured values. It might make an interesting hint that we put "b"
before "a" in the condition. You could also interpret "b" as "before" and
"a" as "after", but also "a" as "arguments".

@capture(lambda a: {"some_identifier": some_func(a.some_argument.some_attr)})
@post(lambda b, a, result: b.some_identifier > result +
a.another_argument.another_attr)
def some_func(some_argument: SomeClass, another_argument:
AnotherClass) -> SomeResult:
...

"b" can be omitted if it is not used. Under the hub, all the arguments to
the condition would be passed by keywords.

In case of inheritance, captures would be inherited as well. Hence the
library would check at run-time that the returned dictionary with captured
values has no identifier that has been already captured, and the linter
checks that statically, before running the code. Reading values captured in
the parent at the code of the child class might be a bit hard -- but that
is case with any inherited methods/properties. In documentation, I'd list
all the captures of both ancestor and the current class.

I'm looking forward to reading your opinion on this and alternative
suggestions :)
Marko

On Tue, 25 Sep 2018 at 18:12, Franklin? Lee 
wrote:

> On Sun, Sep 23, 2018 at 2:05 AM Marko Ristin-Kaufmann
>  wrote:
> >
> > Hi,
> >
> > (I'd like to fork from a previous thread, "Pre-conditions and
> post-conditions", since it got long and we started discussing a couple of
> different things. Let's discuss in this thread the implementation of a
> library for design-by-contract and how to push it forward to hopefully add
> it to the standard library one day.)
> >
> > For those unfamiliar with contracts and current state of the discussion
> in the previous thread, here's a short summary. The discussion started by
> me inquiring about the possibility to add design-by-contract concepts into
> the core language. The idea was rejected by the participants mainly because
> they thought that the merit of the feature does not merit its costs. This
> is quite debatable and seems to reflect many a discussion about
> design-by-contract in general. Please see the other thread, "Why is
> design-by-contract not widely adopted?" if you are interested in that
> debate.
> >
> > We (a colleague of mine and I) decided to implement a library to bring
> design-by-contract to Python since we don't believe that the concept will
> make it into the core language anytime soon and we needed badly a tool to
> facilitate our work with a growing code base.
> >
> > The library is av

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Hugh,

> As soon as you need to document your code, and
> > this is what most modules have to do in teams of more than one person
> > (especially so if you are developing a library for a wider audience), you
> > need to write down the contracts. Please see above where I tried to
> > explained  that 2-5) are inferior approaches to documenting contracts
> > compared to 1).
>
> You left off option 6), plain text. Comments. Docstrings.
>

 That was actually the option 2):

> 2) Write precondtions and postconditions in docstring of the method as
> human text.


The problem with text is that it is not verifiable and hence starts to
"rot". Noticing that text is wrong involves much more developer time &
attention than automatically verifying the formal contracts.

In Python we can write something like:

def foo(x):
x.bar(y)

What's the type of x? What's the type of y? What is the contract of bar?
Don't know, don't care. x, or y, can be an instance, a class, a module, a
proxy for a remote web service. The only "contract" is that object x will
respond to message bar that takes one argument. Object x, do whatever
you want with it.

I still don't see how this is connected to contracts or how contracts play
a role there? If foo can accept any x and return any result then there is *no
*contract. But hardly any function is like that. Most exercise a certain
behavior on a subset of possible input values. The outputs also  satisfy
certain contracts, *i.e.* they also live in a certain subset of possible
outputs. (Please mind that I don't mean strictly numerical ranges here --
it can be any subset of structured data.) As I already mentioned, the
contracts have nothing to do with typing. You can use them for runtime type
checks -- but that's a reduction of the concept to a very particular use
case. Usually contracts read like this (from the numpy example linked in
another message,
https://www.numpy.org/devdocs/reference/generated/numpy.ndarray.transpose.html#numpy.ndarray.transpose
):

ndarray.transpose(**axes*)

Returns a view of the array with axes transposed.

*For a 1-D array, this has no effect. (To change between column and row
vectors, first cast the 1-D array into a matrix object.) For a 2-D array,
this is the usual matrix transpose. For an n-D array, if axes are given,
their order indicates how the axes are permuted (see Examples). If axes are
not provided and a.shape = (i[0], i[1], ... i[n-2], i[n-1]), then
a.transpose().shape = (i[n-1], i[n-2], ... i[1], i[0]).*
(emphasis mine)

Mind the three postconditions (case 1D array, case 2D array, case N-D
array).

As for 4) reading the code, why not? "Use the source, Luke" is now a
> programming cliche because it works. It's particularly appropriate for
> Python packages which are usually distributed in source form and, as
> you yourself noted, easy to read.


Because it is hard and costs a lot of time. The point of encapsulating a
function is that I as a user don't have to know its details of
implementation and its wider dependencies in the implementation. Looking at
the code is the tool of last resort to figure out the contracts. Imagine if
you had to look at the implementation of numpy.transpose() to figure out
what happens when transposing a N-D array.

Cheers,
Marko

On Tue, 25 Sep 2018 at 13:13, Hugh Fisher  wrote:

> > Date: Mon, 24 Sep 2018 09:46:16 +0200
> > From: Marko Ristin-Kaufmann 
> > To: Python-Ideas 
> > Subject: Re: [Python-ideas] Why is design-by-contracts not widely
> > adopted?
> > Message-ID:
> >  i7eq2gr8kddofoneab-qvt8lz...@mail.gmail.com>
> > Content-Type: text/plain; charset="utf-8"
>
> [munch]
>
> > Their users would hugely benefit from a more mature
> > and standardized contracts library with informative violation messages.
>
> Will respond in another message, because it's a big topic.
>
> > I really don't see how DbC has to do with duck typing (unless you reduce
> it
> > to mere isinstance conditions, which would simply be a straw-man
> argument)
> > -- could you please clarify?
>
> I argue that Design by Contract doesn't make sense for Python and other
> dynamically typed, duck typed languages because it's contrary to how the
> language, and the programmer, expects to work.
>
> In Python we can write something like:
>
> def foo(x):
> x.bar(y)
>
> What's the type of x? What's the type of y? What is the contract of bar?
> Don't know, don't care. x, or y, can be an instance, a class, a module, a
> proxy for a remote web service. The only "contract" is that object x will
> respond to message bar that takes one argument. Object x, do whatever
> you want with it.
>
> And that's a feature, not a bug, not bad design. It follows Postel's Law
> for Internet p

Re: [Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

Hi James and Franklin,

getsource() definitely does not work. I tried for a long, long time to make
it work and finally gave up. I parse in icontract the whole file where the
lambda function resides and use asttokens to locate the node of the lambda
(along some tree traversing upwards and making assumptions where the
condition lambda lives).

Have a look at:

https://github.com/Parquery/icontract/blob/391d43005287831892b19dfdcbcfd3d48662c638/icontract/represent.py#L309

and
https://github.com/Parquery/icontract/blob/391d43005287831892b19dfdcbcfd3d48662c638/icontract/represent.py#L157


On Tue, 25 Sep 2018 at 19:48, James Lu  wrote:

> > I'm surprised you haven't found
> >inspect.getsource(func)
>
> I did. That’s exactly what I was describing in the paragraph. It wouldn’t
> work in interactive mode and it includes everything on the same line of the
> lambda definition.
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely

[Marko Ristin-Kaufmann]

Hi Hugh,

Software projects, in any language, never have enough time to do everything.
> By your own example, the Python developers of numpy, OpenCV, nlk, and
> sklearn; *who most certainly weren't writing contracts;* produced better
> quality
> software than the Eiffel equivalent developers who (I assume) did use DbC.
> Shouldn't the Eiffel developers be changing their development method, not
> the Python developers?

(emphasis mine)

This is *absolutely* *not true* as you can notice if you multiply any two
matrices of wrong dimensions in numpy or opencv (or use them as weights in
sklearn).

For example, have a look at OpenCV functions. *Most of them include
preconditions and postconditions *(*e.g., *
https://docs.opencv.org/3.4.3/dc/d8c/namespacecvflann.html#a57191110b01f200e478c658f3b7a362d).
I would even go as far to claim that OpenCV would be unusable without the
contracts. Imagine if you had to figure out the dimensions of the matrix
after each operation if it were lacking in the documentation. That would
make the development sluggish as a snail.

Numpy provides contracts in text, *e.g. *see
https://www.numpy.org/devdocs/reference/generated/numpy.ndarray.transpose.html#numpy.ndarray.transpose
:
ndarray.transpose(**axes*)

Returns a view of the array with axes transposed.

For a 1-D array, this has no effect. (To change between column and row
vectors, first cast the 1-D array into a matrix object.) For a 2-D array,
this is the usual matrix transpose. For an n-D array, if axes are given,
their order indicates how the axes are permuted (see Examples). If axes are
not provided and a.shape = (i[0], i[1], ... i[n-2], i[n-1]), then
a.transpose().shape = (i[n-1], i[n-2], ... i[1], i[0]).

As you can see, there are three contracts: 1) no effect on 1D array, 2) if
a 2D array, it equals the matrix transpose, 3) if n-D array, the order of
axes indicates the permutation. The contract 3) is written out formally. It
might not be very clear or precise what is meant in 2) where formalizing it
(at least to a certain extent) would remove many doubts.

It is obvious to me that supplementing or replacing these contracts *in
text* with *formal *contracts (1 and 2, since 3 is already formal) is
extremely beneficial since: a) many developers use numpy and an improvement
in documentation (such as higher precision and clarity) has a large impact
on the users and b) enforcing the contracts automatically (be it only
during testing or in production) prevents bugs related to contract
violation in numpy such that the users can effectively rely on the
contracts. The argument b) is important since now I just rely that these
statements are true whenever I use numpy. If there is an error in numpy it
takes a long time to figure out since I doubt the last that there is an
error in numpy and especially it takes even longer that I suspect numpy of
not satisfying its written contracts.

Please mind that contracts can be toggled on/off whenever the performance
is important so that slow execution is not an argument against the formal
contracts.

Cheers,
Marko

On Tue, 25 Sep 2018 at 14:01, Hugh Fisher  wrote:

> > Date: Mon, 24 Sep 2018 09:46:16 +0200
> > From: Marko Ristin-Kaufmann 
> > To: Python-Ideas 
> > Subject: Re: [Python-ideas] Why is design-by-contracts not widely
> > adopted?
>
> [munch]
>
> > Python is easier to write and read, and there are no libraries which are
> > close in quality in Eiffel space (notably, Numpy, OpenCV, nltk and
> > sklearn). I really don't see how the quality of these libraries have
> > anything to do with lack (or presence) of the contracts. OpenCV and Numpy
> > have contracts all over their code (written as assertions and not
> > documented), albeit with very non-informative violation messages. And
> they
> > are great libraries. Their users would hugely benefit from a more mature
> > and standardized contracts library with informative violation messages.
>
> I would say the most likely outcome of adding Design by Contract would
> be no change in the quality or usefulness of these libraries, with a small
> but not insignificant chance of a decline in quality.
>
> Fred Brooks in his "No Silver Bullet" paper distinguished between essential
> complexity, which is the problem we try to solve with software, and
> accidental
> complexity, solving the problems caused by your tools and/or process that
> get in the way of solving the actual problem. "Yak shaving" is a similar,
> less
> formal term for accidental complexity, when you have to do something before
> you can do something before you can actually do some useful work.
>
> Adding new syntax or semantics to a programming language very often adds
> accidental complexity.
>
> C and Python (currently) are known as simple languages. When starting a
> programming proj

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Steve,
I'll give it a shot and implement a proof-of-concept icontrac-macro library
based on macropy and see if that works. I'll keep you posted.

Cheers,
Marko


On Tue, 25 Sep 2018 at 12:08, Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Marko Ristin-Kaufmann writes:
>
>  > Thanks a lot for pointing us to macropy -- I was not aware of the
> library,
>  > it looks very interesting!
>  >
>  > Do you have any experience how macropy fit
>
> Sorry, no.  I was speaking as someone who is familiar with macros from
> Lisp but doesn't miss them in Python, and who also has been watching
> python-dev and python-ideas for about two decades now, so I've heard
> of things like MacroPy and know how the core developers think to a
> great extent.
>
>  > I'm also a bit worried how macropy would work out in the libraries
>  > published to pypi -- imagine if many people start using contracts.
>  > Suddenly, all these libraries would not only depend on a contract
> library
>  > but on a macro library as well.
>
> That's right.
>
>  > Is that something we should care about?
>
> Yes.  Most Pythonistas (at least at present) don't much like macros.
> They fear turning every program into its own domain-specific language.
> I can't claim much experience with dependency hell, but I think that's
> much less important from your point of view (see below).
>
> My point is mainly that, as you probably are becoming painfully aware,
> getting syntax changes into Python is a fairly drawnout process.  For
> an example of the kind of presentation that motivates people to change
> their mind from the default state of "if it isn't in Python yet,
> YAGNI" to "yes, let's do *this* one", see
> https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings
>
> Warning: Tim Peters is legendary, though still active occasionally.
> All he has to do is post to get people to take notice.  But this
> Appendix is an example of why he gets that kind of R-E-S-P-E-C-T.[1]
>
> So the whole thing is a secret plot ;-) to present the most beautiful
> syntax possible in your PEP (which will *not* be about DbC, but rather
> about a small set of enabling syntax changes, hopefully a singleton),
> along with an extended example, or a representative sample, of usage.
> Because you have a working implementation using MacroPy (or the less
> pretty[2] but fewer dependencies version based on condition strings
> and eval) people can actually try it on their own code and (you hope,
> they don't :-) they find a nestful of bugs by using it.
>
>  > Potential dependency hell? (I already have a bad feeling about
>  > making icontract depend on asttokens and considerin-lining
>  > asttokens into icontract particularly for that reason).
>
> I don't think so.  First, inlining an existing library is almost
> always a bad idea.  As for the main point, if the user sticks to one
> major revision, and only upgrades to compatible bugfixes in the
> Python+stdlib distribution, I don't see why two or three libraries
> would be a major issue for a feature that the developer/project uses
> extremely frequently.  I've rarely experienced dependency hell, and in
> both cases it was web frameworks (Django and Zope, to be specific, and
> the dependencies involved were more or less internal to those
> frameworks).  If you or people you trust have other experience, forget
> what I just said. :-)
>
> Of course it depends on the library, but as long as the library is
> pretty strict about backward compatibility, you can upgrade it and get
> new functionality for other callers in your code base (which are
> likely to appear, you know -- human beings cannot stand to leave a
> tool unused once they install it!)
>
>  > > Note that this means *you cannot use macros in a file that is run
>  > > directly*, as it will not be passed through the import hooks.
>  >
>  > That would make contracts unusable in any stand-alone script,
>  > right?
>
> Yes, but really, no:
>
> # The run.py described by the MacroPy docs assumes a script that
> # runs by just importing it.  I don't have time to work out
> # whether that makes more sense.  This idiom of importing just a
> # couple of libraries, and then invoking a function with a
> # conventional name such as "run" or "process" is quite common.
> # If you have docutils install, check out rstpep2html.py.
>
> import macropy.activate
> from my_contractful_library import main
> main()
>
> and away you go.  5 years from now that script will be a badge of
> honor among Pythonic DbCers, and you won't be willing to give it up!
>

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

to enhance/extended
> documentation.


It is up to the developer to decide which contracts are enforced during
testing, production or displayed in the documentation (you can pick the
subset of the three, it's not an exclusion). This feature ("enabled"
argument to a contract) has been already implemented in the icontract
library.

Some of the examples you've been giving would be better expressed with
a more capable type system in my view (e.g. Rust's), but I have no
good idea about adding that into Python  :/.

I don't see how type system would help regardless how strict it would be?
Unless *each *input and *each *output represent a special type, which would
be super confusing as soon as you would put them in the containers and have
to struggle with invariance, contravariance and covariance. Please see
https://github.com/rust-lang/rfcs/issues/1077 for a discussion about
introducing DbC to Rust. Unfortunately, the discussion about contracts in
Rust is also based on misconceptions (*e.g., *see
https://github.com/rust-lang/rfcs/issues/1077#issuecomment-94582917) --
there seems to be something wrong in the way anybody proposing DbC exposes
contracts to the wider audience and miss to address these issues in a good
way. So most people just react instinctively with "80% already covered with
type systems" / "mere runtime type checks, use assert" and "that's only an
extension to testing, so why bother" :(.

I would now like to answer Hugh and withdraw from the discussion pro/contra
formal contracts unless there is a rational, logical argument disputing the
DbC in its entirety (not in one of its specific aspects or as a
misconception/straw-man). A lot has been already said, many articles have
been written (I linked some of the pages which I thought were short & good
reads and I would gladly supply more reading material). I doubt I can find
a better way to contribute to the discussion.

Cheers,
Marko


On Tue, 25 Sep 2018 at 10:01, Robert Collins 
wrote:

> On Mon, 24 Sep 2018 at 19:47, Marko Ristin-Kaufmann
>  wrote:
> >
> > Hi,
> >
> > Thank you for your replies, Hugh and David! Please let me address the
> points in serial.
> >
> > Obvious benefits
> > You both seem to misconceive the contracts. The goal of the
> design-by-contract is not reduced to testing the correctness of the code,
> as I reiterated already a couple of times in the previous thread. The
> contracts document formally what the caller and the callee expect and need
> to satisfy when using a method, a function or a class. This is meant for a
> module that is used by multiple people which are not necessarily familiar
> with the code. They are not a niche. There are 150K projects on pypi.org.
> Each one of them would benefit if annotated with the contracts.
>
> You'll lose folks attention very quickly when you try to tell folk
> what they do and don't understand.
>
> Claiming that DbC annotations will improve the documentation of every
> single library on PyPI is an extraordinary claim, and such claims
> require extraordinary proof.
>
> I can think of many libraries where necessary pre and post conditions
> (such as 'self is still locked') are going to be noisy, and at risk of
> reducing comprehension if the DbC checks are used to enhance/extended
> documentation.
>
> Some of the examples you've been giving would be better expressed with
> a more capable type system in my view (e.g. Rust's), but I have no
> good idea about adding that into Python  :/.
>
> Anyhow, the thing I value most about python is its pithyness: its
> extremely compact, allowing great developer efficiency, but the cost
> of testing is indeed excessive if the tests are not structured well.
> That said, its possible to run test suites with 10's of thousands of
> tests in only a few seconds, so there's plenty of headroom for most
> projects.
>
> -Rob
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Steve and others,
After some thinking, I'm coming to a conclusion that it might be wrong to
focus too much about how the contracts are written -- as long as they are
formal, easily  transformable to another representation and fairly
maintainable.

Whether it's with a lambda, without, with "args" or "a", with "old" or "o"
-- it does not matter that much as long as it is pragmatic and not
something crazy complex. This would also mean that we should not add
complexity (*e.g., *by adding macros) and limit the magic as much as
possible.

It is actually much more important in which form they are presented to the
end-user. I already made an example with sphinx-icontract in a message
before -- an improved version might use mathematical symbols (*e.g., *replace
all() with ∀, replace len() with |.|, nicely use subscripts for ranges, use
case distinction with curly bracket "{" instead of if.. else ..., etc.).
This would make them even shorter and easier to parse. Let me iterate the
example I already pasted in the thread before to highlight what I have in
mind:
packagery.resolve_initial_paths(*initial_paths*)

Resolve the initial paths of the dependency graph by recursively adding *.py
files beneath given directories.
Parameters:

*initial_paths* (List[Path]) – initial paths as absolute paths
Return type:

List[Path]
Returns:

list of initial files (*i.e.* no directories)
Requires:

   - all(pth.is_absolute() for pth in initial_paths)

Ensures:

   - all(pth in result for pth in initial_paths if pth.is_file()) (Initial
   files also in result)
   - len(result) >= len(initial_paths) if initial_paths else result == []
   - all(pth.is_absolute() for pth in result)
   - all(pth.is_file() for pth in result)


The contracts need to extend __doc__ of the function accordingly (and the
contracts in __doc__ also need to reflect the inheritance of the
contracts!), so that we can use help().

There should be also a plugin for Pycharm, Pydev, vim and emacs to show the
contracts in an abbreviated and more readable form in the code and only
show them in raw form when we want to edit them (*i.e., *when we move
cursor over them). I suppose inheritance of contracts needs to be reflected
in quick-inspection windows, but not in the code view.

Diffs and github/bitbucket/... code reviews might be a bit cumbersome since
they enforce the raw form of the contracts, but as long as syntax is
pragmatic, I don't expect this to be a blocker.

Is this a sane focus?

Cheers,
Marko

On Tue, 25 Sep 2018 at 08:18, Marko Ristin-Kaufmann 
wrote:

> Hi Steve,
> Thanks a lot for pointing us to macropy -- I was not aware of the library,
> it looks very interesting!
>
> Do you have any experience how macropy fit with current IDEs and static
> linters (pylint, mypy)? I fired up pylint and mypy on the sample code from
> their web site, played a bit with it and it seems that they go along well.
>
> I'm also a bit worried how macropy would work out in the libraries
> published to pypi -- imagine if many people start using contracts.
> Suddenly, all these libraries would not only depend on a contract library
> but on a macro library as well. Is that something we should care about?
> Potential dependency hell? (I already have a bad feeling about making
> icontract depend on asttokens and considerin-lining asttokens into
> icontract particularly for that reason).
>
> I'm also worried about this one (from
> https://macropy3.readthedocs.io/en/latest/overview.html):
>
>> Note that this means *you cannot use macros in a file that is run
>> directly*, as it will not be passed through the import hooks.
>
>
> That would make contracts unusable in any stand-alone script, right?
>
> Cheers,
> Marko
>
> On Tue, 25 Sep 2018 at 06:56, Stephen J. Turnbull <
> turnbull.stephen...@u.tsukuba.ac.jp> wrote:
>
>> Barry Scott writes:
>>
>>  > > @requires(lambda self, a, o: self.sum == o.sum - a.amount)
>>  > > def withdraw(amount: int) -> None:
>>  > > ...
>>  > >
>>  > > There is this lambda keyword in front, but it's not too bad?
>>  >
>>  > The lambda smells of internals that I should not have to care about
>>  > being exposed.
>>  > So -1 on lambda being required.
>>
>> If you want to get rid of the lambda you can use strings and then
>> 'eval' them in the condition.  Adds overhead.
>>
>> If you want to avoid the extra runtime overhead of parsing
>> expressions, it might be nice to prototype with MacroPy.  This should
>> also allow eliminating the lambda by folding it into the macro (I
>> haven't used MacroPy but it got really good reviews by fans of that
>> kind of thing).  It would be possible to avoid decorator syntax if you
>> want to with this 

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Steve,
Thanks a lot for pointing us to macropy -- I was not aware of the library,
it looks very interesting!

Do you have any experience how macropy fit with current IDEs and static
linters (pylint, mypy)? I fired up pylint and mypy on the sample code from
their web site, played a bit with it and it seems that they go along well.

I'm also a bit worried how macropy would work out in the libraries
published to pypi -- imagine if many people start using contracts.
Suddenly, all these libraries would not only depend on a contract library
but on a macro library as well. Is that something we should care about?
Potential dependency hell? (I already have a bad feeling about making
icontract depend on asttokens and considerin-lining asttokens into
icontract particularly for that reason).

I'm also worried about this one (from
https://macropy3.readthedocs.io/en/latest/overview.html):

> Note that this means *you cannot use macros in a file that is run
> directly*, as it will not be passed through the import hooks.


That would make contracts unusable in any stand-alone script, right?

Cheers,
Marko

On Tue, 25 Sep 2018 at 06:56, Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Barry Scott writes:
>
>  > > @requires(lambda self, a, o: self.sum == o.sum - a.amount)
>  > > def withdraw(amount: int) -> None:
>  > > ...
>  > >
>  > > There is this lambda keyword in front, but it's not too bad?
>  >
>  > The lambda smells of internals that I should not have to care about
>  > being exposed.
>  > So -1 on lambda being required.
>
> If you want to get rid of the lambda you can use strings and then
> 'eval' them in the condition.  Adds overhead.
>
> If you want to avoid the extra runtime overhead of parsing
> expressions, it might be nice to prototype with MacroPy.  This should
> also allow eliminating the lambda by folding it into the macro (I
> haven't used MacroPy but it got really good reviews by fans of that
> kind of thing).  It would be possible to avoid decorator syntax if you
> want to with this implementation.
>
> I'm not sure that DbC is enough of a fit for Python that it's worth
> changing syntax to enable nice syntax natively, but detailed reports
> on a whole library (as long as it's not tiny) using DbC with a nice
> syntax (MacroPy would be cleaner, but I think it would be easy to "see
> through" the quoted conditions in an eval-based implementation) would
> go a long way to making me sit up and take notice.  (I'm not
> influential enough to care about, but I suspect some committers would
> be impressed too.  YMMV)
>
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi Barry,
I think the main issue with pyffel is that it can not support function
calls in general. If I understood it right, and Angus please correct me,
you would need to wrap every function that you would call from within the
contract.

But the syntax is much nicer than icontract or dpcontracts (see these
packages on pypi). What if we renamed "args" argument and "old" argument in
those libraries to just "a" and "o", respectively? Maybe that gives
readable code without too much noise:

@requires(lambda self, a, o: self.sum == o.sum - a.amount)
def withdraw(amount: int) -> None:
...

There is this lambda keyword in front, but it's not too bad?

I'll try to contact dpcontracts maintainers. Maybe it's possible to at
least merge a couple of libraries into one and make it a de facto standard.
@Agnus, would you also like to join the effort?

Cheers,
Marko





Le lun. 24 sept. 2018 à 19:57, Barry Scott  a
écrit :

>
>
> On 23 Sep 2018, at 11:13, Angus Hollands  wrote:
>
> Hi Marko,
>
> I think there are several ways to approach this problem, though am not
> weighing in on whether DbC is a good thing in Python. I wrote a simple
> implementation of DbC which is currently a run-time checker. You could,
> with the appropriate tooling, validate statically too (as with all
> approaches). In my approach, I use a “proxy” object to allow the contract
> code to be defined at function definition time. It does mean that some
> things are not as pretty as one would like - anything that cannot be hooked
> into with magic methods i.e isinstance, but I think this is acceptable as
> it makes features like old easier. Also, one hopes that it encourages
> simpler contract checks as a side-effect. Feel free to take a look -
> https://github.com/agoose77/pyffel
> It is by no means well written, but a fun PoC nonetheless.
>
> This is an interesting PoC, nice work! I like that its easy to read the
> tests.
>
> Given a library like this the need to build DbC into python seems
> unnecessary.
>
> What do other people think?
>
> Barry
>
>
>
> Regards,
> Angus
> ​
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

of such
a tool. Again, please do correct me and contradict -- I don't want to sound
condescending or arrogant -- I literally can't wrap my head around why
*anybody* would dispense of a such an easy-to-use tool that gives you
better documentation (*i.e. *superior to approaches 2-5) except for lack of
formal skills and lack of supporting library. If you think that the
documentation is *not *important, then please, do explain that since it
goes counter to all my previous experience and intuition (which, of course,
can be wrong).

*Why not Eiffel?*
Hugh wrote:

> Secondly, these "obvious" benefits. If they're obvious, I want to know why
> aren't you using Eiffel? It's a programming language designed around DbC
> concepts. It's been around for three decades, at least as long as Python or
> longer. There's an existing base of compilers and support tools and
> libraries
> and textbooks and experienced programmers to work with.
>
> Could it be that Python has better libraries, is faster to develop for,
> attracts
> more programmers? If so, I suggest it's worth considering that this might
> be *because* Python doesn't have DbC.


Python is easier to write and read, and there are no libraries which are
close in quality in Eiffel space (notably, Numpy, OpenCV, nltk and
sklearn). I really don't see how the quality of these libraries have
anything to do with lack (or presence) of the contracts. OpenCV and Numpy
have contracts all over their code (written as assertions and not
documented), albeit with very non-informative violation messages. And they
are great libraries. Their users would hugely benefit from a more mature
and standardized contracts library with informative violation messages.


*Duck Typing*
Hugh wrote:

> And I wouldn't use DbC for Python because
> I wouldn't find it helpful for the kind of dynamic, exploratory development
> I do in Python. I don't write strict contracts for Python code because in a
> dynamically typed, and duck typed, programming language they just don't
> make sense to me. Which is not to say I think Design by Contract is bad,
> just that it isn't good for Python.
>

I really don't see how DbC has to do with duck typing (unless you reduce it
to mere isinstance conditions, which would simply be a straw-man argument)
-- could you please clarify? As soon as you need to document your code, and
this is what most modules have to do in teams of more than one person
(especially so if you are developing a library for a wider audience), you
need to write down the contracts. Please see above where I tried to
explained  that 2-5) are inferior approaches to documenting contracts
compared to 1).

As I wrote above, I would be very, very thankful if you point me to other
approaches (apart from 1-5) that are superior to contracts or state an
argument why approaches 2-5) are superior to the contracts since that is
what I miss to see.

Cheers,
Marko


On Sun, 23 Sep 2018 at 12:34, Hugh Fisher  wrote:

> > Date: Sun, 23 Sep 2018 07:09:37 +0200
> > From: Marko Ristin-Kaufmann 
> > To: Python-Ideas 
> > Subject: [Python-ideas] Why is design-by-contracts not widely adopted?
>
> [ munch ]
>
> > *. *After properly reading about design-by-contract and getting deeper
> into
> > the topic, there is no rational argument against it and the benefits are
> > obvious. And still, people just wave their hand and continue without
> > formalizing the contracts in the code and keep on writing them in the
> > descriptions.
>
> Firstly, I see a difference between rational arguments against Design By
> Contract (DbC) and against DbC in Python. Rejecting DbC for Python is
> not the same as rejecting DbC entirely.
>
> Programming languages are different, obviously. Python is not the same
> as C is not the same as Lisp... To me this also means that different
> languages are used for different problem domains, and in different styles
> of development. I wouldn't use DbC in programming C or assembler
> because it's not really helpful for the kind of low level close to the
> machine
> stuff I use C or assembler for. And I wouldn't use DbC for Python because
> I wouldn't find it helpful for the kind of dynamic, exploratory development
> I do in Python. I don't write strict contracts for Python code because in a
> dynamically typed, and duck typed, programming language they just don't
> make sense to me. Which is not to say I think Design by Contract is bad,
> just that it isn't good for Python.
>
> Secondly, these "obvious" benefits. If they're obvious, I want to know why
> aren't you using Eiffel? It's a programming language designed around DbC
> concepts. It's been around for three decades, at least as long as Python or
> longer. There's an existing base of compilers and support tools and
> libraries
> and textboo

[Python-ideas] "old" values in postconditions

[Marko Ristin-Kaufmann]

Hi,

(I'd like to fork from a previous thread, "Pre-conditions and
post-conditions", since it got long and we started discussing a couple of
different things. Let's discuss in this thread the implementation of a
library for design-by-contract and how to push it forward to hopefully add
it to the standard library one day.)

For those unfamiliar with contracts and current state of the discussion in
the previous thread, here's a short summary. The discussion started by me
inquiring about the possibility to add design-by-contract concepts into the
core language. The idea was rejected by the participants mainly because
they thought that the merit of the feature does not merit its costs. This
is quite debatable and seems to reflect many a discussion about
design-by-contract in general. Please see the other thread, "Why is
design-by-contract not widely adopted?" if you are interested in that
debate.

We (a colleague of mine and I) decided to implement a library to bring
design-by-contract to Python since we don't believe that the concept will
make it into the core language anytime soon and we needed badly a tool to
facilitate our work with a growing code base.

The library is available at http://github.com/Parquery/icontract. The hope
is to polish it so that the wider community could use it and once the
quality is high enough, make a proposal to add it to the standard Python
libraries. We do need a standard library for contracts, otherwise projects
with *conflicting* contract libraries can not integrate (*e.g., *the
contracts can not be inherited between two different contract libraries).

So far, the most important bits have been implemented in icontract:

   - Preconditions, postconditions, class invariants
   - Inheritance of the contracts (including strengthening and weakening of
   the inherited contracts)
   - Informative violation messages (including information about the values
   involved in the contract condition)
   - Sphinx extension to include contracts in the automatically generated
   documentation (sphinx-icontract)
   - Linter to statically check that the arguments of the conditions are
   correct (pyicontract-lint)

We are successfully using it in our code base and have been quite happy
about the implementation so far.

There is one bit still missing: accessing "old" values in the postcondition
(*i.e., *shallow copies of the values prior to the execution of the
function). This feature is necessary in order to allow us to verify state
transitions.

For example, consider a new dictionary class that has "get" and "put"
methods:

from typing import Optional

from icontract import post

class NovelDict:
def length(self)->int:
...

def get(self, key: str) -> Optional[str]:
...

@post(lambda self, key, value: self.get(key) == value)
@post(lambda self, key: old(self.get(key)) is None and
old(self.length()) + 1 == self.length(),
  "length increased with a new key")
@post(lambda self, key: old(self.get(key)) is not None and
old(self.length()) == self.length(),
  "length stable with an existing key")
def put(self, key: str, value: str) -> None:
...

How could we possible implement this "old" function?

Here is my suggestion. I'd introduce a decorator "before" that would allow
you to store whatever values in a dictionary object "old" (*i.e. *an object
whose properties correspond to the key/value pairs). The "old" is then
passed to the condition. Here is it in code:

# omitted contracts for brevity
class NovelDict:
def length(self)->int:
...

# omitted contracts for brevity
def get(self, key: str) -> Optional[str]:
...

@before(lambda self, key: {"length": self.length(), "get": self.get(key)})
@post(lambda self, key, value: self.get(key) == value)
@post(lambda self, key, old: old.get is None and old.length + 1 ==
self.length(),
  "length increased with a new key")
@post(lambda self, key, old: old.get is not None and old.length ==
self.length(),
  "length stable with an existing key")
def put(self, key: str, value: str) -> None:
...

The linter would statically check that all attributes accessed in "old"
have to be defined in the decorator "before" so that attribute errors would
be caught early. The current implementation of the linter is fast enough to
be run at save time so such errors should usually not happen with a
properly set IDE.

"before" decorator would also have "enabled" property, so that you can turn
it off (*e.g., *if you only want to run a postcondition in testing). The
"before" decorators can be stacked so that you can also have a more
fine-grained control when each one of them is running (some during test,
some during test and in production). The linter would enforce that before's
"enabled" is a disjunction of all the "enabled"'s of the corresponding
postconditions where the old value appears.

Is this a sane approach to "old" values? Any alternative 

[Python-ideas] Why is design-by-contracts not widely adopted?

[Marko Ristin-Kaufmann]

Hi,

(I'd like to fork from a previous thread, "Pre-conditions and
post-conditions", since it got long and we started discussing a couple of
different things. Let's put the general discussion related to
design-by-contract in this thread and I'll spawn another thread for the
discussion about the concrete implementation of a design-by-contract
library in Python.)

After the discussion we had on the list and after browsing the internet a
bit, I'm still puzzled why design-by-contract was not more widely adopted
and why so few languages support it. Please have a look at these articles
and answers:

   - https://www.leadingagile.com/2018/05/design-by-contract-part-one/
   -
   
https://ask.slashdot.org/story/07/03/10/009237/why-is-design-by-contract-not-more-popular
   -- this one is from 2007, but represents well IMO the way people discuss it
   -
   
https://stackoverflow.com/questions/481312/why-is-design-by-contract-not-so-popular-compared-to-test-driven-development
   and this answer in particular
   https://stackoverflow.com/a/28680756/1600678
   -
   
https://softwareengineering.stackexchange.com/questions/128717/why-is-there-such-limited-support-for-design-by-contract-in-most-modern-programm


I did see that there are a lot of misconceptions about it ("simple
asserts", "developer overhead", "needs upfront design", "same as unit
testing"). This is probably the case with any novel concept that people are
not familiar with. However, what does puzzle me is that once the
misconceptions are rectified ("it's not simple asserts", "the development
is actually faster", "no need for upfront design", "not orthogonal, but dbc
+ unit testing is better than just unit testing"), the concept is still
discarded

*. *After properly reading about design-by-contract and getting deeper into
the topic, there is no rational argument against it and the benefits are
obvious. And still, people just wave their hand and continue without
formalizing the contracts in the code and keep on writing them in the
descriptions.

* Why is that so? *I'm completely at loss about that -- especially about
the historical reasons (some mentioned that design-by-contract did not take
off since Bertrand Meyer holds the trademark on the term and because of his
character. Is that the reason?).

One explanation that seems plausible to me is that many programmers are
actually having a hard time with formalization and logic rules (*e.g.,
*implication,
quantifiers), maybe due to missing education (*e.g. *many programmers are
people who came to programming from other less-formal fields). It's hence
easier for them to write in human text and takes substantial cognitive load
to formalize these thoughts in code. Does that explains it?

What do you think? What is the missing part of the puzzle?

Cheers,
Marko
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi,
I implemented a sphinx extension to include contracts in the documentation:
https://github.com/Parquery/sphinx-icontract

The extension supports inheritance. It lists all the postconditions and
invariants including the inherited one. The preconditions are grouped by
classes with ":requires:" and ":requires else:".

I was unable to get the syntax highlighting for in-line code to work --
does anybody know how to do that in Sphinx?

The results can be seen, *e.g.* in this documentation:
https://pypackagery.readthedocs.io/en/latest/packagery.html

On a more general note: is there any blocker left why you would *not *use
the contracts in your code? Anything I could improve or fix in icontract
that would make it more convincing to use (apart from implementing static
contract checking and automatic test generation :))?

Cheers,
Marko



On Thu, 20 Sep 2018 at 22:52, Marko Ristin-Kaufmann 
wrote:

> Hi,
> Again a brief update.
>
> * icontract supports now static and class methods (thanks to my colleague
> Adam Radomski) which came very handy when defining a group of functions as
> an interface *via* an abstract (stateless) class. The implementors then
> need to all satisfy the contracts without needing to re-write them. You
> could implement the same behavior with *_impl or _* ("protected") methods
> where public methods would add the contracts as asserts, but we find the
> contracts-as-decorators more elegant (N functions instead of 2*N; see the
> snippet below).
>
> * We implemented a linter to statically check that the contract arguments
> are defined correctly. It is available as a separate Pypi package
> pyicontract-lint (https://github.com/Parquery/pyicontract-lint/). Next
> step will be to use asteroid to infer that the return type of the condition
> function is boolean. Does it make sense to include PEX in the release on
> github?
>
> * We plan to implement a sphinx plugin so that contracts can be readily
> visible in the documentation. Is there any guideline or standard/preferred
> approach how you would expect this plugin to be implemented? My colleagues
> and I don't have any experience with sphinx plugins, so any guidance is
> very welcome.
>
> class Component(abc.ABC, icontract.DBC):
> """Initialize a single component."""
>
> @staticmethod
> @abc.abstractmethod
> def user() -> str:
> """
> Get the user name.
>
> :return: user which executes this component.
> """
> pass
>
> @staticmethod
> @abc.abstractmethod
> @icontract.post(lambda result: result in groups())
> def primary_group() -> str:
> """
> Get the primary group.
>
> :return: primary group of this component
> """
> pass
>
> @staticmethod
> @abc.abstractmethod
> @icontract.post(lambda result: result.issubset(groups()))
> def secondary_groups() -> Set[str]:
> """
> Get the secondary groups.
>
> :return: list of secondary groups
> """
> pass
>
> @staticmethod
> @abc.abstractmethod
> @icontract.post(lambda result: all(not pth.is_absolute() for pth in 
> result))
> def bin_paths(config: mapried.config.Config) -> List[pathlib.Path]:
> """
> Get list of binary paths used by this component.
>
> :param config: of the instance
> :return: list of paths to binaries used by this component
> """
> pass
>
> @staticmethod
> @abc.abstractmethod
> @icontract.post(lambda result: all(not pth.is_absolute() for pth in 
> result))
> def py_paths(config: mapried.config.Config) -> List[pathlib.Path]:
> """
> Get list of py paths used by this component.
>
> :param config: of the instance
> :return: list of paths to python executables used by this component
> """
> pass
>
> @staticmethod
> @abc.abstractmethod
> @icontract.post(lambda result: all(not pth.is_absolute() for pth in 
> result))
> def dirs(config: mapried.config.Config) -> List[pathlib.Path]:
> """
> Get directories used by this component.
>
> :param config: of the instance
> :return: list of paths to directories used by this component
> """
> pass
>
>
> On Sat, 15 Sep 2018 at 22:14, Marko Ristin-Kaufmann <
> marko.ris...@gmail.com> wrote:
>
>> Hi David Maertz and Michael Lee,
&g

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi,
Again a brief update.

* icontract supports now static and class methods (thanks to my colleague
Adam Radomski) which came very handy when defining a group of functions as
an interface *via* an abstract (stateless) class. The implementors then
need to all satisfy the contracts without needing to re-write them. You
could implement the same behavior with *_impl or _* ("protected") methods
where public methods would add the contracts as asserts, but we find the
contracts-as-decorators more elegant (N functions instead of 2*N; see the
snippet below).

* We implemented a linter to statically check that the contract arguments
are defined correctly. It is available as a separate Pypi package
pyicontract-lint (https://github.com/Parquery/pyicontract-lint/). Next step
will be to use asteroid to infer that the return type of the condition
function is boolean. Does it make sense to include PEX in the release on
github?

* We plan to implement a sphinx plugin so that contracts can be readily
visible in the documentation. Is there any guideline or standard/preferred
approach how you would expect this plugin to be implemented? My colleagues
and I don't have any experience with sphinx plugins, so any guidance is
very welcome.

class Component(abc.ABC, icontract.DBC):
"""Initialize a single component."""

@staticmethod
@abc.abstractmethod
def user() -> str:
"""
Get the user name.

:return: user which executes this component.
"""
pass

@staticmethod
@abc.abstractmethod
@icontract.post(lambda result: result in groups())
def primary_group() -> str:
"""
Get the primary group.

:return: primary group of this component
"""
pass

@staticmethod
@abc.abstractmethod
@icontract.post(lambda result: result.issubset(groups()))
def secondary_groups() -> Set[str]:
"""
Get the secondary groups.

:return: list of secondary groups
"""
pass

@staticmethod
@abc.abstractmethod
@icontract.post(lambda result: all(not pth.is_absolute() for pth in result))
def bin_paths(config: mapried.config.Config) -> List[pathlib.Path]:
"""
Get list of binary paths used by this component.

:param config: of the instance
:return: list of paths to binaries used by this component
"""
pass

@staticmethod
@abc.abstractmethod
@icontract.post(lambda result: all(not pth.is_absolute() for pth in result))
def py_paths(config: mapried.config.Config) -> List[pathlib.Path]:
"""
Get list of py paths used by this component.

:param config: of the instance
:return: list of paths to python executables used by this component
"""
pass

@staticmethod
@abc.abstractmethod
@icontract.post(lambda result: all(not pth.is_absolute() for pth in result))
def dirs(config: mapried.config.Config) -> List[pathlib.Path]:
"""
    Get directories used by this component.

:param config: of the instance
:return: list of paths to directories used by this component
"""
pass


On Sat, 15 Sep 2018 at 22:14, Marko Ristin-Kaufmann 
wrote:

> Hi David Maertz and Michael Lee,
>
> Thank you for raising the points. Please let me respond to your comments
> in separation. Please let me know if I missed or misunderstood anything.
>
> *Assertions versus contracts.* David wrote:
>
>> I'm afraid that in reading the examples provided it is difficulties for
>> me not simply to think that EVERY SINGLE ONE of them would be FAR easier to
>> read if it were an `assert` instead.
>>
>
> I think there are two misunderstandings on the role of the contracts.
> First, they are part of the function signature, and not of the
> implementation. In contrast, the assertions are part of the implementation
> and are completely obscured in the signature. To see the contracts of a
> function or a class written as assertions, you need to visually inspect the
> implementation. The contracts are instead engraved in the signature and
> immediately visible. For example, you can test the distinction by pressing
> Ctrl + q in Pycharm.
>
> Second, assertions are only suitable for preconditions. Postconditions are
> practically unmaintainable as assertions as soon as you have multiple early
> returns in a function. The invariants implemented as assertions are always
> unmaintainable in practice (except for very, very small classes) -- you
> need to inspect each function of the class and all their return statements
> and manually add assertions for each invariant.

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

 else. I agree those
> kinds of libraries can be useful, but I don't think they're necessarily
> useful enough to be part of the standard library or to be a technique
> Python programmers should automatically use by default.
>
> What might be interesting is somebody wrote a library that does something
> more then just adding asserts. For example, one idea might be to try
> hooking up a contracts library to hypothesis (or any other library that
> does quickcheck-style testing). That might be a good way of partially
> addressing the problems up above -- you write out your invariants, and a
> testing library extracts that information and uses it to automatically
> synthesize interesting test cases.
>
> (And of course, what would be very cool is if the contracts could be
> verified statically like you can do in languages like dafny -- that way,
> you genuinely would be able to avoid writing many kinds of tests and could
> have confidence your contracts are upheld. But I understanding implementing
> such verifiers are extremely challenging and would probably have too-steep
> of a learning curve to be usable by most people anyways.)
>
> -- Michael
>
>
>
> On Fri, Sep 14, 2018 at 11:51 PM, Marko Ristin-Kaufmann <
> marko.ris...@gmail.com> wrote:
>
>> Hi,
>> Let me make a couple of practical examples from the work-in-progress (
>> https://github.com/Parquery/pypackagery, branch mristin/initial-version)
>> to illustrate again the usefulness of the contracts and why they are, in my
>> opinion, superior to assertions and unit tests.
>>
>> What follows is a list of function signatures decorated with contracts
>> from pypackagery library preceded by a human-readable description of the
>> contracts.
>>
>> The invariants tell us what format to expect from the related string
>> properties.
>>
>> @icontract.inv(lambda self: self.name.strip() == self.name)
>> @icontract.inv(lambda self: self.line.endswith("\n"))
>> class Requirement:
>> """Represent a requirement in requirements.txt."""
>>
>> def __init__(self, name: str, line: str) -> None:
>> """
>> Initialize.
>>
>> :param name: package name
>> :param line: line in the requirements.txt file
>> """
>> ...
>>
>> The postcondition tells us that the resulting map keys the values on
>> their name property.
>>
>> @icontract.post(lambda result: all(val.name == key for key, val in 
>> result.items()))
>> def parse_requirements(text: str, filename: str = '') -> 
>> Mapping[str, Requirement]:
>> """
>> Parse requirements file and return package name -> package requirement 
>> as in requirements.txt
>>
>> :param text: content of the ``requirements.txt``
>> :param filename: where we got the ``requirements.txt`` from (URL or path)
>> :return: name of the requirement (*i.e.* pip package) -> parsed 
>> requirement
>> """
>> ...
>>
>>
>> The postcondition ensures that the resulting list contains only unique
>> elements. Mind that if you returned a set, the order would have been lost.
>>
>> @icontract.post(lambda result: len(result) == len(set(result)), 
>> enabled=icontract.SLOW)
>> def missing_requirements(module_to_requirement: Mapping[str, str],
>>  requirements: Mapping[str, Requirement]) -> 
>> List[str]:
>> """
>> List requirements from module_to_requirement missing in the 
>> ``requirements``.
>>
>> :param module_to_requirement: parsed ``module_to_requiremnt.tsv``
>> :param requirements: parsed ``requirements.txt``
>> :return: list of requirement names
>> """
>> ...
>>
>> Here is a bit more complex example.
>> - The precondition A requires that all the supplied relative paths
>> (rel_paths) are indeed relative (as opposed to absolute).
>> - The postcondition B ensures that the initial set of paths (given in
>> rel_paths) is included in the results.
>> - The postcondition C ensures that the requirements in the results are
>> the subset of the given requirements.
>> - The precondition D requires that there are no missing requirements (*i.e.
>> *that each requirement in the given module_to_requirement is also
>> defined in the given requirements).
>>
>> @icontract.pre(lambda rel_paths: all(rel_pth.root == "" for rel_pth in 
>> rel_paths))  # A
>> @icontract.post

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi,
Let me make a couple of practical examples from the work-in-progress (
https://github.com/Parquery/pypackagery, branch mristin/initial-version) to
illustrate again the usefulness of the contracts and why they are, in my
opinion, superior to assertions and unit tests.

What follows is a list of function signatures decorated with contracts from
pypackagery library preceded by a human-readable description of the
contracts.

The invariants tell us what format to expect from the related string
properties.

@icontract.inv(lambda self: self.name.strip() == self.name)
@icontract.inv(lambda self: self.line.endswith("\n"))
class Requirement:
"""Represent a requirement in requirements.txt."""

def __init__(self, name: str, line: str) -> None:
"""
Initialize.

:param name: package name
:param line: line in the requirements.txt file
"""
...

The postcondition tells us that the resulting map keys the values on their
name property.

@icontract.post(lambda result: all(val.name == key for key, val in
result.items()))
def parse_requirements(text: str, filename: str = '') ->
Mapping[str, Requirement]:
"""
Parse requirements file and return package name -> package
requirement as in requirements.txt

:param text: content of the ``requirements.txt``
:param filename: where we got the ``requirements.txt`` from (URL or path)
:return: name of the requirement (*i.e.* pip package) -> parsed requirement
"""
...


The postcondition ensures that the resulting list contains only unique
elements. Mind that if you returned a set, the order would have been lost.

@icontract.post(lambda result: len(result) == len(set(result)),
enabled=icontract.SLOW)
def missing_requirements(module_to_requirement: Mapping[str, str],
 requirements: Mapping[str, Requirement]) -> List[str]:
"""
List requirements from module_to_requirement missing in the
``requirements``.

:param module_to_requirement: parsed ``module_to_requiremnt.tsv``
:param requirements: parsed ``requirements.txt``
:return: list of requirement names
"""
...

Here is a bit more complex example.
- The precondition A requires that all the supplied relative paths
(rel_paths) are indeed relative (as opposed to absolute).
- The postcondition B ensures that the initial set of paths (given in
rel_paths) is included in the results.
- The postcondition C ensures that the requirements in the results are the
subset of the given requirements.
- The precondition D requires that there are no missing requirements (*i.e.
*that each requirement in the given module_to_requirement is also defined
in the given requirements).

@icontract.pre(lambda rel_paths: all(rel_pth.root == "" for rel_pth in
rel_paths))  # A
@icontract.post(
lambda rel_paths, result: all(pth in result.rel_paths for pth in rel_paths),
enabled=icontract.SLOW,
description="Initial relative paths included")  # B
@icontract.post(
lambda requirements, result: all(req.name in requirements for req
in result.requirements),
enabled=icontract.SLOW)  # C
@icontract.pre(
lambda requirements, module_to_requirement:
missing_requirements(module_to_requirement, requirements) == [],
enabled=icontract.SLOW)  # D
def collect_dependency_graph(root_dir: pathlib.Path, rel_paths:
List[pathlib.Path],
 requirements: Mapping[str, Requirement],
 module_to_requirement: Mapping[str, str])
-> Package:

"""
Collect the dependency graph of the initial set of python files
from the code base.

:param root_dir: root directory of the codebase such as
"/home/marko/workspace/pqry/production/src/py"
:param rel_paths: initial set of python files that we want to
package. These paths are relative to root_dir.
:param requirements: requirements of the whole code base, mapped
by package name
:param module_to_requirement: module to requirement correspondence
of the whole code base
:return: resolved depedendency graph including the given initial
relative paths,
"""

I hope these examples convince you (at least a little bit :-)) that
contracts are easier and clearer to write than asserts. As noted before in
this thread, you can have the same *behavior* with asserts as long as you
don't need to inherit the contracts. But the contract decorators make it
very explicit what conditions should hold *without* having to look into the
implementation. Moreover, it is very hard to ensure the postconditions with
asserts as soon as you have a complex control flow since you would need to
duplicate the assert at every return statement. (You could implement a
context manager that ensures the postconditions, but a context manager is
not more readable than decorators and you have to duplicate them as
documentation in the docstring).

In my view, contracts are also superior to many kinds of tests. As the
contracts are *always* enforced, they also enforce 

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi,
A brief follow-up (latest version 1.5.3): I removed the dependency on meta
package so that now all comprehensions and generator expressions work. I
still had to depend on asttokens in order to get the source code of the
condition function. Is there maybe an alternative solution which uses only
standard libraries?

Any thoughts or feedback on the icontract library in general?

Cheers,
Marko

On Mon, 10 Sep 2018 at 09:29, Marko Ristin-Kaufmann 
wrote:

> Hi,
>
> I implemented the inheritance via meta classes and function and class
> attributes for pre/postconditions and invariants, respectively. Unless I
> missed something, this is as far as we can go without the proper language
> support with a library based on decorators:
> https://github.com/Parquery/icontract (version 1.5.0)
>
>  Note that it is actually a complete implementation of design-by-contract
> that supports both weakening of the preconditions and strengthening of the
> postconditions and invariants.
>
> Could you please have a look and let me know what you think about the
> current implementation?
>
> Once we are sure that there is nothing obvious missing, I'd like to move
> forward and discuss whether we could add this library (or rewrite it) into
> the standard Python libraries and what needs to be all fixed till to make
> it that far.
>
> Cheers,
> Marko
>
> On Sat, 8 Sep 2018 at 21:34, Jonathan Fine  wrote:
>
>> Michel Desmoulin wrote:
>>
>> > Isn't the purpose of "assert" to be able to do design by contract ?
>> >
>> > assert test, "error message is the test fail"
>> >
>> > I mean, you just write your test, dev get a feedback on problems, and
>> > prod can remove all assert using -o.
>> >
>> > What more do you need ?
>>
>> Good question. My opinion is that assert statements are good. I like them.
>>
>> But wait, more is possible. Here are some ideas.
>>
>> 1. Checking the return value (or exception). This is a post-condition.
>>
>> 2. Checking return value, knowing the input values. This is a more
>> sophisticated post-condition.
>>
>> 3. Adding checks around an untrusted function - possibly third party,
>> possibly written in C.
>>
>> 4. Selective turning on and off of checking.
>>
>> The last two, selective checks around untrusted functions, I find
>> particularly interesting.
>>
>> Suppose you have a solid, trusted, well-tested and reliable system.
>> And you add, or change, a function called wibble(). In this situation,
>> errors are most likely to be in wibble(), or in the interface to
>> wibble().
>>
>> So which checks are most valuable? I suggest the answer is
>>
>> 1. Checks internal to wibble.
>>
>> 2. Pre-conditions and post-conditions for wibble
>>
>> 3. Pre-conditions for any function called by wibble.
>>
>> Suppose wibble calls wobble. We should certainly have the system check
>> wobble's preconditions, in this situation. But we don't need wobble to
>> run checks all the time. Only when the immediate caller is wibble.
>>
>> I think assertions and design-by-contract point in similar directions.
>> But design-by-contract takes you further, and is I suspect more
>> valuable when the system being built is large.
>>
>> Thank you, Michel, for your good question.
>>
>> --
>> Jonathan
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi,

I implemented the inheritance via meta classes and function and class
attributes for pre/postconditions and invariants, respectively. Unless I
missed something, this is as far as we can go without the proper language
support with a library based on decorators:
https://github.com/Parquery/icontract (version 1.5.0)

 Note that it is actually a complete implementation of design-by-contract
that supports both weakening of the preconditions and strengthening of the
postconditions and invariants.

Could you please have a look and let me know what you think about the
current implementation?

Once we are sure that there is nothing obvious missing, I'd like to move
forward and discuss whether we could add this library (or rewrite it) into
the standard Python libraries and what needs to be all fixed till to make
it that far.

Cheers,
Marko

On Sat, 8 Sep 2018 at 21:34, Jonathan Fine  wrote:

> Michel Desmoulin wrote:
>
> > Isn't the purpose of "assert" to be able to do design by contract ?
> >
> > assert test, "error message is the test fail"
> >
> > I mean, you just write your test, dev get a feedback on problems, and
> > prod can remove all assert using -o.
> >
> > What more do you need ?
>
> Good question. My opinion is that assert statements are good. I like them.
>
> But wait, more is possible. Here are some ideas.
>
> 1. Checking the return value (or exception). This is a post-condition.
>
> 2. Checking return value, knowing the input values. This is a more
> sophisticated post-condition.
>
> 3. Adding checks around an untrusted function - possibly third party,
> possibly written in C.
>
> 4. Selective turning on and off of checking.
>
> The last two, selective checks around untrusted functions, I find
> particularly interesting.
>
> Suppose you have a solid, trusted, well-tested and reliable system.
> And you add, or change, a function called wibble(). In this situation,
> errors are most likely to be in wibble(), or in the interface to
> wibble().
>
> So which checks are most valuable? I suggest the answer is
>
> 1. Checks internal to wibble.
>
> 2. Pre-conditions and post-conditions for wibble
>
> 3. Pre-conditions for any function called by wibble.
>
> Suppose wibble calls wobble. We should certainly have the system check
> wobble's preconditions, in this situation. But we don't need wobble to
> run checks all the time. Only when the immediate caller is wibble.
>
> I think assertions and design-by-contract point in similar directions.
> But design-by-contract takes you further, and is I suspect more
> valuable when the system being built is large.
>
> Thank you, Michel, for your good question.
>
> --
> Jonathan
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi Ethan,
You are right, I deleted it without noticing.

It should say: pre(len(lst) < 10).


Le jeu. 30 août 2018 à 23:02, Ethan Furman  a écrit :

> On 08/30/2018 01:49 PM, Marko Ristin-Kaufmann wrote:
>
> > classC(A):
> >  # C.some_func also inherits the contracts from A.
> >  #   It weakens the precondition:
> >  #   it operates either on sorted lists OR
> >  #   the lists that are shorter than 10 elements.
> >  #
> >  #  It strenghthens the postcondition:
> >  #   It needs to return an integer larger than
> >  #   the length of the input list AND
> >  #   the result needs to be divisible by 2.
> >  @icontract.post(lambdaresult: result %2==0)
> >  defsome_func(self, lst: List[int]) ->int:
> >  # ...
>
> I think you forgot an @icontract.pre() here.
>
> --
> ~Ethan~
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi,

@David Mertz, Eric Fahlgren re inheritance: thank you very much for your
suggestions. I will try to see how inheritance can be implemented with
metaclasses and annotations and put it into icontract library.

@David Mertz re costs:

> Adding a new feature, even if it is *technically* backwards compatible,
> has HUGE costs.
>
[...]

> It's a specific feature that is relatively well defined as a concept (and
> precisely defined in regard to Eiffel specifically; but we might not
> implement those *exact* semantics).  It's also a feature that no languages
> in particularly widespread use have decided to implement at the language
> level.  I've chatted with Meyer; he's definitely very smart and definitely
> strongly opinionated, but *I also think he's wrong about the overall
> importance of this feature* *versus lots of others* [emphasis Marko].
>
> In my mind, this feature doesn't come close to meeting the burden of those
> high costs listed above (and others I did not mention).  But I don't have
> any say in what the core developers will do, beyond in that they might be
> influenced by my opinion here.
>

I am also aware of the costs; when I wrote "useful", I actually meant
"useful and meaningful to implement given the costs that you mention.
Please apologize for my inexact wording (I write these emails in the
evening after work).

Related to the text I emphasized, would you mind to explain a bit more
in-depth which features you have in mind? I see contracts formally written
out and automatically verified as a completely indispensable tool in large
projects with multiple people involved. As I already outlined, writing them
in documentation leads to code rot. Not writing assumptions about the data
structures at all causes, in my opinion, tons of problems and very, very
slow development since each programmer working on the legacy code needs to
figure these assumptions over and over again. While testing is not
substitute for thinking, I possible don't see how you can add/refactor code
in big projects and consider all the interactions between the components.
Unit testing can get you only as far, since you always test only for a
concrete case. Contracts get you much further, not to mention the
automatically generated tests which I can't conceive practical without some
sort of contracts in the code. I have also to admit that I was schooled
with contracts so I might have a distorted view on the matter (Bertrand
Meyer taught Introduction to Programming and all the software engineering
university classes; I also worked as a teaching assistant with him for the
Software Architecture class).

I will try to see how far we can implement contracts as a library.
Personally, I would prefer dedicated syntax and I do believe that dedicated
syntax merits the costs since these constructs would lead to much better
programming in the Python world.

If we can't get new syntax, having contracts in the standard library would
do it for me as a compromise -- it would standardize, even if in ugly way
(as Steven D'Aprano pointed out), how we deal with contracts in Python and
would allow for integration into IDEs and static analysis tools.

On Thu, 30 Aug 2018 at 05:01, Ethan Furman  wrote:

> On 08/29/2018 04:53 PM, Hugh Fisher wrote:
> > From: Marko Ristin-Kaufmann:
>
> >> There seems to be evidence that design-by-contract is useful. Let me
> cite
> >> Bertrand Meyer from his article "Why not program right?" that I already
> >> mentioned before:
> >
> > I don't think that being useful by itself should be enough. I think new
> features
> > should also be "Pythonic" and I don't see design by contract notation as
> a
> > good fit.
>
> I don't see type annotation notation as a good fit, either, and yet we
> have it.  Seems to me that DbC would be just as
> useful as type annotations (and I find D'Aprano's example syntax very
> readable).
>
> > For design by contract, as others have noted Python assert statements
> > work fine for simple preconditions and postconditions.
>
> And print statements work fine for simple debugging.  New features are not
> added for the simple cases, but the complex,
> or non-obviously correct, or the very useful cases.
>
> > Yes there's more to design by contract than simple assertions, but it's
> not
> > just adding syntax. Meyer often uses the special "old" construct in his
> post
> > condition examples, a trivial example being
> >
> >  ensure count = old.count + 1
>
> I can see where that could get expensive quickly.
>
> > How do we do that in Python? And another part of design by contract (at
> > least according to Meyer) is that it's not enough to just raise an
> exception,
> > but there must be a guarantee that it is handled and

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi,
To clarify the benefits of the contracts, let me give you an example from
our code base:

@icontract.pre(lambda x: x >= 0)
@icontract.pre(lambda y: y >= 0)
@icontract.pre(lambda width: width >= 0)
@icontract.pre(lambda height: height >= 0)
@icontract.pre(lambda x, width, img: x + width <= pqry.opencv.width_of(img))
@icontract.pre(lambda y, height, img: y + height <= pqry.opencv.height_of(img))
@icontract.post(lambda self: (self.x, self.y) in self)
@icontract.post(lambda self: (self.x + self.width - 1, self.y +
self.height - 1) in self)
@icontract.post(lambda self: (self.x + self.width, self.y +
self.height) not in self)
def __init__(self, img: np.ndarray, x: int, y: int, width: int,
height: int) -> None:
self.img = img[y:y + height, x:x + width].copy()
self.x = x
self.y = y
self.width = width
self.height = height

def __contains__(self, xy: Tuple[int, int]) -> bool:
x, y = xy
return self.x <= x < self.x + self.width and \
   self.y <= y < self.y + self.height

We use mypy and type annotations for typing, and we don't need contracts
for that. In this particular case, contracts are very handy to formulate
how we deal with pixels. If you carefully look at the contracts, you will
see:
* pixels are indexed starting from 0 (as oppose to starting from 1)
* region-of-interest needs to fit within an image. It can not be outside of
its boundaries
* pixels within the region-of-interest are in [x, x + width), [y, y +
height) (where "[" means inclusive and ")" exclusive)

*Why not types?* These constraints are either impossible or very hard to
write as types. Moreover, I doubt that you could give such types meaningful
names, and I expect the readability to suffer immensely. I suppose any
constraints involving more than a couple of variables is hard to write as a
type.

*Benefits.* You could write all of this in human-readable informal
docstring of a class, but then it would never be verified. This way we are
sure that contracts are always there. Whenever we test, we can be sure that
all the contracted properties hold. Mind that we do not test that they hold
for that single test case -- we automatically test that they hold for all
the test cases.

This is, in my opinion, far superior to ambiguous human documentation or
print statements (which are deleted after the debugging). I suppose *any*
larger code base built by a team of more than one developer needs this kind
of rigor. The contracts are not meant to be used only in high-risk
applications -- they are meant to improve any code base.

*Problem without standard support. *The current libraries (dpcontracts,
icontract) can deal with pre and postconditions and class invariants of a
concrete class.

However, the current libraries break as soon as you have inheritance. There
is no way in python to inherit the function decorators and to modify them.
If we are to inherit from the above-mentioned class ROI, we need to make
sure that the invariants of the parent class hold (*e.g., *what is
contained in a ROI) as well as invariants of the child class. We might want
to weaken the requirements (*e.g., *a ROI that can deal with pixels outside
of an image) such that x and y can be an arbitrary numbers, not restricted
to 0 <= x < img.width and 0 <= y < img.height respectively.

Additionally, without standard approach, we do not know how to deal with
contracts when we have a third-party tool that would like to use them (*e.g.,
*for automatic generation of unit tests, static testing or visualization in
IDE or in documentation).

@Wes Turner : If I understood you correctly, you are
looking for a library that gives you verbose messages when a contract is
breached. Please have a look at icontract library:
https://github.com/Parquery/icontract

We added informative messages particularly because we wanted to have
verbose output when something goes wrong.This was helpful not only during
the development, but also in production since failure cases were hard to
reproduce and anticipate in the first place (otherwise, they wouldn't have
made it into the production). Here is an example with an error message:

>>> class B:... def __init__(self) -> None:... self.x = 7.. 
>>> def y(self) -> int:... return 2.. def __repr__(self) -> 
>>> str:... return "instance of B"...>>> class A:... def 
>>> __init__(self)->None:... self.b = B().. def __repr__(self) 
>>> -> str:... return "instance of A"...>>> SOME_GLOBAL_VAR = 13>>> 
>>> @icontract.pre(lambda a: a.b.x + a.b.y() > SOME_GLOBAL_VAR)... def 
>>> some_func(a: A) -> None:... pass...>>> an_a = A()>>> some_func(an_a)
Traceback (most recent call last):
  ...
icontract.ViolationError: Precondition violated: (a.b.x + a.b.y()) >
SOME_GLOBAL_VAR:SOME_GLOBAL_VAR was 13
a was instance of A
a.b was instance of B
a.b.x was 7
a.b.y() was 2


On Tue, 28 Aug 2018 at 03:19, Wes Turner  wrote:

> Thanks for the explanation.
>
> This may be a bit OT,
> 

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi,

@Paul Moore: thanks for pointing out that many people are not familiar with
design-by-contract. I was not aware of that.

Let me give you a very short introduction into contracts and what they are
good for. I'll review some existing libraries and highlight what features
we missed (and why we developed our own library). I will finally conclude
with why all these solutions (including our own one) are not a replacement
for a proper support of design-by-contract in the language.

Design-by-Contract
Design-by-contract was gradually introduced as a concept by Bertrand Meyer
in the 1980ies to provide a formal and verifiable interface specification
between the software components. Up to then the interfaces were usually
defined formally in terms of abstract data types (think of records /
structs and classes). He extended abstract data types by adding "contract"
conditions that should hold at different points during the execution of a
program. The contracts allow you to formally write down your expectations
about the program (as opposed to writing it informally in documentation of
a class or a function). Hence we can automatically test that they hold
either statically or during the execution of the program. This gives us
many-fold benefits:

   - contracts prevent code rot (since they can be checked by a compiler or
   a static analysis tool such as mypy),
   - allow us to have much more sophisticated automatic generation of unit
   tests and automatic formal proofs of a program,
   - make the documentation explicit, accurate and verifiable and
   - accelerate the development since errors are caught early.

The contracts are categorized as follows depending on at which point in the
program they are verified:

   - Preconditions are contracts that should hold before the execution of a
   function. The *caller* is responsible to fulfill the preconditions.
   - Postconditions are contracts that should hold after the execution of a
   function. The *callee* is responsible to fulfill the postconditions.
   - Invariants should hold throughout the execution of the program. There
   are two types of invariants: loop invariants and class invariants.
   - Loop invariants should hold before and after each iteration step of a
   loop.
   - Class invariants should hold throughout the life time of a class (
   *i.e.* between the calls to the public methods). The class invariants
   are suspended during the construction of an instance and in private methods
   to avoid cycles. You can think of the constructor method and public methods
   being responsible to fulfill the class invariants.

The concept of design-by-contract is not limited only to concrete classes,
but can be also applied to class hierarchies. Preconditions, postconditions
and invariants are inherited. They can be also modified in the following
ways:

   - The child class needs to fulfill all the invariants of its antecedent
   classes and its own ones.
   - The preconditions of a function of a child class can "weaken" or
   "relax" the preconditions of the parent class. In other words, it needs to
   fulfill *either* the preconditions of the parent class *or *its own set
   of preconditions. This is reflected in Eiffel by using the keyword *require
   else.*
   - The postconditions of a  a child class can "strengthen" or "tighten"
   the postconditions of the parent class. The function needs to fulfill all
   the postconditions of the parent class' function *and *its own set of
   postconditions. In Eiffel, this is designated with *ensure then* keyword.

Invariants operate only on the values of instance properties. Preconditions
operate on both function arguments and instance properties. Postconditions
need yet one more instrument: they operate on function arguments, instance
properties and the result of a function, but can access all these values
both at their *old* state, before the function call, and their *new *state,
after the function call. In Eiffel, you use the keyword *old* to indicate
the value before the function call.

Let me illustrate the concepts by adapting the examples from
https://www.eiffel.org/doc/eiffel/ET-_Design_by_Contract_%28tm%29%2C_Assertions_and_Exceptions
and https://www.eiffel.org/doc/eiffel/ET-_Inheritance. I will paraphrase
the example code in Python assuming that invariant, require, ensure and old
were introduced as keywords.

*class *Account:
*def *__init__(self, balance: int) -> *None*:
self.balance = balance
self.deposits = []  *# type: List[int]

**def *update(self, sum: int)->*None*:
require: sum >= 0

self.balance += sum
self.deposits.append(sum)

ensure: len(self.deposits) = len(old self.deposits) + 1, *"one
more deposit"
*ensure: self.balance = old self.balance + sum, *"updated"

*invariant: *not *self.deposits *or *self.balance ==
sum(self.deposits), *"consistent balance"
*invariant: self.deposits *or *self.balance == 0, *"zero if no deposits"

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi,

I had a look at the messages related to the PEP in question (PEP 316) in
the archive. As far as I can tell, the main objection is that you can
achieve contracts by implementing it with decorators.

I think that these objections miss what actually Daniel Moisset wrote in
his message: contracts are more than pre- and post-condition checks on a
function. The inheritance of decorators does not imply just inheriting the
pre- and post-conditions, but also relaxing and tightening them
(represented by "require else" and "ensure then" in Eiffel). If this is to
be used effectively in practice with little overhead then we would either
need to introduce new syntax to the language or make the compiler improve
the byte code on the fly.

Is there any chance to introduce these constructs in the language or is it
too small a feature for such a big change?

Since we wanted to have contracts in Golang, we implemented a tool that
synchronizes the documentation of a function with the function code (
https://github.com/Parquery/gocontracts). Maybe this is the easier path to
follow in Python as well?

@Wes Turner: thanks for pointing to pycontracts. I'm aware of the library.
It implements only contracts based on a single property. We found that
limiting and rolled out our own solution that suited us much better:
https://github.com/Parquery/icontract/

I also found informative messages on contract breach to be particularly
important for fast development and error inspection in the production.

Cheers,
Marko


On 21 August 2018 at 04:44, Wes Turner  wrote:

> pycontracts may be worth a look.
>
> https://andreacensi.github.io/contracts/
>
> - @contract decorator, annotations, docstrings
>
> IDK if pycontracts supports runtime parameter validations that involve
> more than one parameter.
>
> Inheritance does appear to be supported,
> as are numpy array dimension constraints.
>
> I can't recall whether the pycontracts expression language precedes MyPy
> compile-time annotations; both with one syntax really would be great.
>
>
> On Monday, August 20, 2018, Daniel Moisset 
> wrote:
>
>> I think that annotations were suggested because you could write an
>> expression there without getting evaluated.
>>
>> I've thought about this problem many times in the past (as a Python dev
>> with a long history working in Eiffel too) For me one of the crucial
>> issue that is hard to translate into the python model is that the
>> assertions (say, a function precondition) are not conceptually part of the
>> function itself, but the interface of the class. The "natural" python ways
>> of attaching these assertions somehow to the function object do not work
>> when you also use inheritance, because when you override a method the new
>> function object clobbers the previous one. I've experimented at some point
>> on how to put them in classes (and doing metaclass or __getattribute__
>> tricks) but nothing convinced me). In general, the way that python puts
>> method call and inheritance semantic in a specific layout of runtime
>> objects (which in general is really clever) seems to be a bit alien to the
>> DbC idea where the asbtraction/interface of the class is conceptually
>> separate and has independent information wrt to the runtime objects.
>>
>>
>> On 16 August 2018 at 18:49, Marko Ristin-Kaufmann > > wrote:
>>
>>> Hi Jonathan and Paul,
>>> Thank you very much for your suggestions! I will try to contact the
>>> author of the PEP.
>>>
>>> Let me clarify a bit a potential misunderstanding. Please mind that
>>> contracts are not tied to individual variables, but to expressions. Think
>>> of it as defining a lambda which takes as input all the arguments of the
>>> function (and a result variable in case of post-conditions) which always
>>> needs to evaluate to True.
>>>
>>> Cheers,
>>> Marko
>>>
>>> Le jeu. 16 août 2018 à 12:24, Paul Moore  a écrit :
>>>
>>>> On Thu, 16 Aug 2018 at 10:41, Jonathan Fine 
>>>> wrote:
>>>> >
>>>> > Hi Marko
>>>> >
>>>> > Thank you for introducing yourself, and clearly stating your question.
>>>> > That helps us all. You asked:
>>>> >
>>>> > > Could somebody update me on the state of the discussion on this
>>>> matter?
>>>> >
>>>> > I think bring the existing PEP up to date would be a good starting
>>>> > point. Its content hasn't been changed since 2003 (except for PEP-wide
>>>> > admin changes. (Recall that Python 3.0 was released in 2008.)
>>>&g

Re: [Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi Jonathan and Paul,
Thank you very much for your suggestions! I will try to contact the author
of the PEP.

Let me clarify a bit a potential misunderstanding. Please mind that
contracts are not tied to individual variables, but to expressions. Think
of it as defining a lambda which takes as input all the arguments of the
function (and a result variable in case of post-conditions) which always
needs to evaluate to True.

Cheers,
Marko

Le jeu. 16 août 2018 à 12:24, Paul Moore  a écrit :

> On Thu, 16 Aug 2018 at 10:41, Jonathan Fine  wrote:
> >
> > Hi Marko
> >
> > Thank you for introducing yourself, and clearly stating your question.
> > That helps us all. You asked:
> >
> > > Could somebody update me on the state of the discussion on this matter?
> >
> > I think bring the existing PEP up to date would be a good starting
> > point. Its content hasn't been changed since 2003 (except for PEP-wide
> > admin changes. (Recall that Python 3.0 was released in 2008.)
> >
> > https://www.python.org/dev/peps/pep-0316/
> > https://github.com/python/peps/commits/master/pep-0316.txt
> >
> > In fact, revising the PEP might be enough to answer your question.
> > What do you think, Marko?
> >
> > Experts: is there a process for revising old PEPs, such as this one?
> > Or at least a precedent we could follow (or adapt)?
>
> I'm not aware of a formal process, but I'd have thought the following
> steps would be a reasonable approach:
>
> 1. Review the PEP, and research the discussions that happened at the
> time, particularly of interest is why the PEP was deferred.
> 2. Consider what (if anything) has changed since the original deferral
> (which could simply be "time has moved on, people's views may have
> changed" but ideally would include a bit more in the way of concrete
> motivation).
> 3. Contact the original PEP author and ask if he is interested in
> reopening the discussion, collaborating on a revision, or handing the
> PEP over.
> 4. Start up a discussion here, pointing out the original PEP and
> summarising the previous debate and why you want to restart the
> discussion. If you're hoping to change the details of the original
> PEP, summarise your changes and why you feel they are an improvement
> over the original.
>
> To answer the OP's question more directly:
>
> > Could somebody update me on the state of the discussion on this matter?
>
> As far as I am aware, there has been no discussion on this subject
> since the PEP 316 discussions which ended up in its deferral. Elazar
> mentioned PEP 563, and there *may* have been mention of design by
> contract uses in the discussions on that PEP, but you'd have to search
> the mailing list archives to confirm that one way or another.
>
> Hence the suggestions that if you want to restart discussion, reviving
> PEP 316 is likely the best approach.
>
> Paul
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Pre-conditions and post-conditions

[Marko Ristin-Kaufmann]

Hi,

I would be very interested to bring design-by-contract into python 3. I
find design-by-contract particularly interesting and indispensable for
larger projects and automatic generation of unit tests.

I looked at some of the packages found on pypi and also we rolled our own
solution (https://github.com/Parquery/icontract/). I also looked into
https://www.python.org/dev/peps/pep-0316/.

However, all the current solutions seem quite clunky to me. The decorators
involve an unnecessary computational overhead and the implementation of
icontract became quite tricky once we wanted to get the default values of
the decorated function.

Could somebody update me on the state of the discussion on this matter?

I'm very grateful for any feedback on this!
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Hi!

[Marko Ristin-Kaufmann]

Hi,
Please let me briefly introduce myself. I'm Marko and live in Zurich,
Switzerland. I learned Python and Django during an intership in 2007 and
followed its development ever since.

I did research in computer vision (at ETH Zurich) and now I work at
Parquery AG (Zurich, Switzerland) where we implement most of our
infra-structure in Python 3.

Cheers,
Marko
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/