date:20180717


Recently Barry shown an example:

assert len(subdirs := list(path.iterdir())) == 0, subdirs

It looks awful to me. It looks even worse than using asserts for 
validating the user input. The assert has a side effect, and it depends 
on the interpreter option (-O). Even if subdirs is not used outside of 
the assert *now*, it is easy to introduce an error later, and it is hard 
to notice it if tests are not ran with the -O option regularly.


Does PEP 572 encourages writing such code, discourages this, or 
completely forbids?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

2018-07-17 Thread Chris Angelico

On Tue, Jul 17, 2018 at 6:50 PM, Serhiy Storchaka  wrote:
> Recently Barry shown an example:
>
> assert len(subdirs := list(path.iterdir())) == 0, subdirs
>
> It looks awful to me. It looks even worse than using asserts for validating
> the user input. The assert has a side effect, and it depends on the
> interpreter option (-O). Even if subdirs is not used outside of the assert
> *now*, it is easy to introduce an error later, and it is hard to notice it
> if tests are not ran with the -O option regularly.
>
> Does PEP 572 encourages writing such code, discourages this, or completely
> forbids?
>

Asserts with side effects are already a bad idea. PEP 572 makes no
change to this. If you're putting any sort of side effects inside
assertions, you're playing with fire.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

2018-07-17 Thread Antoine Pitrou

On Tue, 17 Jul 2018 18:56:36 +1000
Chris Angelico  wrote:
> On Tue, Jul 17, 2018 at 6:50 PM, Serhiy Storchaka  wrote:
> > Recently Barry shown an example:
> >
> > assert len(subdirs := list(path.iterdir())) == 0, subdirs
> >
> > It looks awful to me. It looks even worse than using asserts for validating
> > the user input. The assert has a side effect, and it depends on the
> > interpreter option (-O). Even if subdirs is not used outside of the assert
> > *now*, it is easy to introduce an error later, and it is hard to notice it
> > if tests are not ran with the -O option regularly.
> >
> > Does PEP 572 encourages writing such code, discourages this, or completely
> > forbids?
> >  
> 
> Asserts with side effects are already a bad idea. PEP 572 makes no
> change to this. If you're putting any sort of side effects inside
> assertions, you're playing with fire.

Well, PEP 572 makes it easier to put side effects in side assertions
since you can now stuff an assignment inside an assert.  Which is what
Serhiy's / Barry's example shows.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Question about PEP 484

2018-07-17 Thread Sebastian Rittau


On 16.07.2018 19:21, Adam Cataldo via Python-Dev wrote:

*

 *

One thing we care about in particular, given the implementation of
pytype, is the detailed definition of what goes in a .pyi file. Do
folks think this makes sense to include as part of PEP 484, or
would this be better in a separate PEP? We’d love to get your
thoughts.

*
It would be useful to define - on a semantic, not syntactic level - what 
constructs a type checker is expected to understand. Not only for 
implementers, but more importantly for stub authors. Sometimes it's hard 
to judge, what constructs type checkers will understand. And while by 
now I think I personally have a solid understanding of what mypy 
understands, I have no idea whether that also applies to pytype, 
PyCharm, or other type checkers.


For example, in one of my first pull requests for typeshed, I tried to 
use the following construct, expecting type checkers to understand it:


    class Foo:
    def bar(self) -> None:
    raise NotImplementedError()

It seems they don't, but mypy understands:

    class Foo:
    @abstractmethod
    def bar(self) -> None: ...

But do I need to import abstractmethod? Would @abc.abstractmethod also 
work? Can I have an assignment "_am = abc.abstractmethod" and then @_am 
would work? Can I alias functions by using assignments in stubs or 
should I use a second function definition? How complex can Python 
version checks be? There are many more of those questions.


If these expectations would be documents, implementers of type checkers 
can still decide not to support certain constructs (or not support them 
yet), or even to support more constructs. But at least such a deviation 
could be documented, so users know what to expect. On the other hand, 
stub authors will know what constructs will likely not work and should 
be avoided.


 - Sebastian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

2018-07-17 Thread Victor Stinner

2018-07-17 10:50 GMT+02:00 Serhiy Storchaka :
> assert len(subdirs := list(path.iterdir())) == 0, subdirs
>
> Does PEP 572 encourages writing such code, discourages this, or completely
> forbids?

If I understood correctly Guido, Python the language must not prevent
developers to experiment various usage of assignement expressions. The
f-string has a similar design: anything even
f'{__import__("os").system("rm -rf /")}' is allowed.

It's more the role of linters like flake8 to warn against "bad practices".

I don't think that PEPs like f-string and assignment expressions must
include *coding styles*.

Sure, f-string and assignment expressions allow to write surprising
and bad code. But you don't them them to write crappy code :-)

I was strongly against f-string at the beginning because I immediately
predicted that people will start to call functions (with side effects)
in f-string... but I didn't see that happen. In fact, developers are
reasonable and use f-string when it's appropriate and safe.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Const access to CPython objects outside of GIL?

2018-07-17 Thread Victor Stinner

2018-07-17 6:18 GMT+02:00 Radim Řehůřek :
> one of our Python projects calls for pretty heavy, low-level optimizations.
>
> We went down the rabbit hole and determined that having access to
> PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict, unicode)
> on Python objects **outside of GIL** might be a good-enough solution. The
> Python objects in question are guaranteed to live and not be mutated
> externally in any way. They're "frozen" and read-only.

IMHO it's a great path to introduce very tricky race conditions in
multithreaded applications! :-)

At the C level, even immutable Python objects are mutable: str, tuple, etc.

IMHO you need a different approach to implement optimizations. For
example, use your objects which don't rely on the GIL to be
consistent. Sadly, I have no experience with that and so cannot
provide any example.

Python releases the GIL *often* and pass pointers to buffers to C
functions. For example, os.stat() writes into a memory block allocated
by Python. But it's a raw memory block, it's much simpler than a
Python object like a tuple.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

On Tue, Jul 17, 2018 at 1:50 AM, Serhiy Storchaka 
wrote:

> Recently Barry shown an example:
>
> assert len(subdirs := list(path.iterdir())) == 0, subdirs
>
> It looks awful to me. It looks even worse than using asserts for
> validating the user input. The assert has a side effect, and it depends on
> the interpreter option (-O). Even if subdirs is not used outside of the
> assert *now*, it is easy to introduce an error later, and it is hard to
> notice it if tests are not ran with the -O option regularly.
>
> Does PEP 572 encourages writing such code, discourages this, or completely
> forbids?
>

The PEP has no specific opinion except it is not forbidden.

Personally I like Barry's example just fine -- assuming `subdirs` is not
used later, this feels like a good use case.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Question about PEP 484

This is a good point. I presume specifying this unambiguously would be a
huge amount of work, and it would mostly codify mypy's current behavior. I
don't think that's within the scope of PEP 484, but it could well be done
as a separate PEP (perhaps an informational one?). I hope you understand
that I am not volunteering.

On Tue, Jul 17, 2018 at 1:51 AM, Sebastian Rittau 
wrote:

> On 16.07.2018 19:21, Adam Cataldo via Python-Dev wrote:
>
>
>
>
> * - One thing we care about in particular, given the implementation of
> pytype, is the detailed definition of what goes in a .pyi file. Do folks
> think this makes sense to include as part of PEP 484, or would this be
> better in a separate PEP? We’d love to get your thoughts. *
>
> It would be useful to define - on a semantic, not syntactic level - what
> constructs a type checker is expected to understand. Not only for
> implementers, but more importantly for stub authors. Sometimes it's hard to
> judge, what constructs type checkers will understand. And while by now I
> think I personally have a solid understanding of what mypy understands, I
> have no idea whether that also applies to pytype, PyCharm, or other type
> checkers.
>
> For example, in one of my first pull requests for typeshed, I tried to use
> the following construct, expecting type checkers to understand it:
>
> class Foo:
> def bar(self) -> None:
> raise NotImplementedError()
>
> It seems they don't, but mypy understands:
>
> class Foo:
> @abstractmethod
> def bar(self) -> None: ...
>
> But do I need to import abstractmethod? Would @abc.abstractmethod also
> work? Can I have an assignment "_am = abc.abstractmethod" and then @_am
> would work? Can I alias functions by using assignments in stubs or should I
> use a second function definition? How complex can Python version checks be?
> There are many more of those questions.
>
> If these expectations would be documents, implementers of type checkers
> can still decide not to support certain constructs (or not support them
> yet), or even to support more constructs. But at least such a deviation
> could be documented, so users know what to expect. On the other hand, stub
> authors will know what constructs will likely not work and should be
> avoided.
>
>  - Sebastian
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Const access to CPython objects outside of GIL?

2018-07-17 Thread Ivan Pozdeev via Python-Dev

On 17.07.2018 7:18, Radim Řehůřek wrote:

Hi all,

one of our Python projects calls for pretty heavy, low-level
optimizations.

We went down the rabbit hole and determined that having access to
PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict,
unicode) on Python objects **outside of GIL** might be a good-enough
solution. The Python objects in question are guaranteed to live and
not be mutated externally in any way. They're "frozen" and read-only.

The standard practice if you need to access something outside of GIL is
to get a private C object from it.
For immutable types, you can get a pointer to the underlying data if the
internal representation is compatible with some C type (e.g. char*
|PyBytes_AsString|(PyObject/ *o/) and FILE*
|PyFile_AsFile|(PyObject/ *p/) (Py2 only -- in Py3, PyFile no longer
wraps stdio FILE) ); otherwise, the C API can produce a copy (e.g.
wchar_t* |PyUnicode_AsWideCharString|(PyObject/ *unicode/,
Py_ssize_t/ *size/) ).

Though you can call whatever you want outside of GIL (it's not like we
can prevent you), anything that's not officially guaranteed you're doing
on your own risk. Even if it happens to work now, it can break in
unpredictable ways at any point in the future.

Under what conditions is it OK to call these 3 functions on such objects?

More generally, what is the CPython 2.7/3.5 contract regarding (lack
of) object mutation, and the need for reference counting and
synchronization via GIL?

Which C API functions are safe to call on "const" objects?

Obviously releasing GIL and then calling C API is hacky, but from
initial experiments, it seems to work (see
https://stackoverflow.com/questions/51351609/can-i-const-access-cpython-objects-without-gil).
But I'm wondering if there's a more formal contract around this behaviour.

Cheers,
Radim

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

--
Regards,
Ivan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert


17.07.18 17:59, Guido van Rossum пише:

The PEP has no specific opinion except it is not forbidden.

Personally I like Barry's example just fine -- assuming `subdirs` is not 
used later, this feels like a good use case.


Shouldn't this problem be solved in the same way as for comprehensions?
Should not the assert statement introduce an implicit lexical scope for 
preventing leaking variables?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

On Tue, Jul 17, 2018 at 8:28 AM, Serhiy Storchaka 
wrote:

> 17.07.18 17:59, Guido van Rossum пише:
>
>> The PEP has no specific opinion except it is not forbidden.
>>
>> Personally I like Barry's example just fine -- assuming `subdirs` is not
>> used later, this feels like a good use case.
>>
>
> Shouldn't this problem be solved in the same way as for comprehensions?
>

No, it's nothing like those.

> Should not the assert statement introduce an implicit lexical scope for
> preventing leaking variables?
>

I don't see why. As Chris said, side effects in asserts are nothing new and
this PEP is not the one to do something about it.

Serhiy, have you even read the latest version of PEP 572? In its current
form it does not introduce any implicit scopes and even goes out of its way
to make assignments in comprehensions "leak" out of the comprehension scope
into the surrounding (explicit, using 'def' or 'lambda') function's scope.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Const access to CPython objects outside of GIL?

[Tim]> There is no intention to support GIL-free access to any Python
objects.  So

> > that's the contract:  "All warranties are null & void if you do just
> about
> > anything while not holding the GIL".
>

[Antoine]

> Actually, accessing a Py_buffer that's been obtained in the regular way

(e.g. with PyObject_GetBuffer) is safe even without the GIL.
>

Same as the docs, I use "Python object" to mean a pointer to PyObject.  In
that sense, a Py_buffer is no more a "Python object" than, e.g,, is a
Py_ssize_t.

If someone wants to muck with the `obj` member of a Py_buffer struct,
_then_ they're back in "Python object" territory (`obj` is a PyObject*) and
so are on their own if they don't obtain the GIL.  Likewise all bets are
off if they don't hold the GIL when calling PyObject_GetBuffer() to begin
with, or PyBuffer_Release() when they're done.

If they want to muck with the `buf` member without the GIL, then if they
care about races among multiple threads mucking with `buf`, they'll have to
supply their own mutual exclusion mechanism.

So your "safe" comes with footnotes too, even though Py_buffer isn't itself
a PyObject*.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Const access to CPython objects outside of GIL?

2018-07-17 Thread Chris Barker via Python-Dev

On Tue, Jul 17, 2018 at 4:34 AM, Victor Stinner  wrote:

>
> IMHO you need a different approach to implement optimizations. For
> example, use your objects which don't rely on the GIL to be
> consistent. Sadly, I have no experience with that and so cannot
> provide any example.
>

I DO NOT understand the technical details, but IIUC, numpy releases the GIL
while doing pure numpy operations.

(and I have experimented with it, and it does seem to be the case -- that
is, you can get enhanced performance with computationally heavy numpy code
an multiple threads)

So maybe some lessons learned there.

The key differences may be that numpy arrays are full-fledged python
objects, but they are not (generally) containers of python objects -- that
is, the actual values are i memory accessed via a C pointer. Whereas a
PyList holds pointers to Python objects, so even if are confident in the
stability of the list itself, the objects in the list may not be stable.

 For example, os.stat() writes into a memory block allocated
> by Python. But it's a raw memory block, it's much simpler than a
> Python object like a tuple.

exactly. -- so is a numpy .data element

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Const access to CPython objects outside of GIL?

2018-07-17 Thread Antoine Pitrou

On Tue, 17 Jul 2018 10:59:02 -0500
Tim Peters  wrote:
> 
> Same as the docs, I use "Python object" to mean a pointer to PyObject.  In
> that sense, a Py_buffer is no more a "Python object" than, e.g,, is a
> Py_ssize_t.

Yes, and the payload of an int object is not a "Python object".
The OP mentioned PyInt_AS_LONG as an example, so clearly the question
was broader than you're painting it to be.

> If they want to muck with the `buf` member without the GIL, then if they
> care about races among multiple threads mucking with `buf`, they'll have to
> supply their own mutual exclusion mechanism.

Define "muck with".  As long as you're reading the memory area(s)
pointed to by the Py_buffer object, you're fine.  If you plan to write
to that memory, obviously you'll need to make sure the various threads
don't overwrite each other, i.e. distribute the work carefully.

But that's a normal provision for multi-threaded algorithms, not a
Python-specific restriction.  I'm assuming someone asking a question
about multi-threaded access to CPython objects already knows all
this :-)

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

2018-07-17 Thread Chris Barker via Python-Dev

>
> I don't see why. As Chris said, side effects in asserts are nothing new
> and
>

Indeed -- this new feature makes it easier to affect the local scope in all
sorts of new places. It was decided that the additional complexity is worth
it to make the language more expressive, and it was also decided not to try
to "lock down" this new feature to only "appropriate" places.

> this PEP is not the one to do something about it.

Hmm --- not sure if it's PEP-worthy, but I'd sure love to see the community
make a strong stance that:

asserts are for tests, and only for tests

Which doesn't necessarily mean that assert statements only exist in test
code (though I personally follow that practice), but it does mean that
assert statements, wherever they live, should not be counted on at run
time, ever.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert


17.07.18 18:48, Guido van Rossum пише:
On Tue, Jul 17, 2018 at 8:28 AM, Serhiy Storchaka > wrote:

Should not the assert statement introduce an implicit lexical scope
for preventing leaking variables?

I don't see why. As Chris said, side effects in asserts are nothing new 
and this PEP is not the one to do something about it.


This side effect is new. No other expressions that can be used in 
asserts leaked local variables before. The only exception is list 
comprehensions in Python 2, and this was fixed in Python 3.


We can't make the assignment expression itself creating its own scope, 
because this will invalidate its purpose. But the problem with assert 
ccould be solved by making assert creating a new lexical scope.


assert expr, msg

could be translated to

if __debug__ and not (lambda: expr)():
raise AssertionError(msg)

instead of

if __debug__ and not expr:
raise AssertionError(msg)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

[Serhiy Storchaka]

> Recently Barry shown an example:
>
>  assert len(subdirs := list(path.iterdir())) == 0, subdirs
>
> It looks awful to me. It looks even worse than using asserts for
> validating the user input. The assert has a side effect, and it depends
> on the interpreter option (-O). Even if subdirs is not used outside of
> the assert *now*, it is easy to introduce an error later, and it is hard
> to notice it if tests are not ran with the -O option regularly.
>


Does PEP 572 encourages writing such code, discourages this, or
> completely forbids?
>

The body of the PEP specifies semantics.  My Appendix A gives some
_opinions_ about "good" and "bad" uses, which boil down to "if it's not
obviously at least a little win, don't use it".

I can't really guess whether the above is an obvious win or not without
context.  It is a win (to my eyes) if the code it replaced was

if __debug__:
subdirs = list(path.iterdir())
assert len(subdirs) == 0, subdirs

in which case the semantics are the same in either spelling, with or
without -O, but the spelling at the top is less annoying ;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert


17.07.18 19:25, Tim Peters пише:

It is a win (to my eyes) if the code it replaced was

     if __debug__:
         subdirs = list(path.iterdir())
         assert len(subdirs) == 0, subdirs

in which case the semantics are the same in either spelling, with or 
without -O, but the spelling at the top is less annoying ;-)


I completely agree.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

2018-07-17 Thread Chris Angelico

On Wed, Jul 18, 2018 at 2:24 AM, Serhiy Storchaka  wrote:
> 17.07.18 18:48, Guido van Rossum пише:
>>
>> On Tue, Jul 17, 2018 at 8:28 AM, Serhiy Storchaka > > wrote:
>> Should not the assert statement introduce an implicit lexical scope
>> for preventing leaking variables?
>>
>> I don't see why. As Chris said, side effects in asserts are nothing new
>> and this PEP is not the one to do something about it.
>
>
> This side effect is new. No other expressions that can be used in asserts
> leaked local variables before. The only exception is list comprehensions in
> Python 2, and this was fixed in Python 3.

There are plenty of expressions that have side effects, though.
Creating local variables is just one type of side effect. If
assertions are being misused, that is the fault of people writing bad
assertions, not of assignment expressions, dict.setdefault(),
subprocess.call(), or any other operation.

Point of interest: You're basically asking for statement-local
assignment, which was one of PEP 572's earliest forms. It received
even more backlash than the current version did. The wheel turns and
the same spoke comes up...

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Const access to CPython objects outside of GIL?

[Tim]

>  > Same as the docs, I use "Python object" to mean a pointer to PyObject.
> In

> that sense, a Py_buffer is no more a "Python object" than, e.g,, is a
> > Py_ssize_t.
>

[Antoine

> Yes, and the payload of an int object is not a "Python object".
> The OP mentioned PyInt_AS_LONG as an example, so clearly the question
> was broader than you're painting it to be.
>

Ah, but I don't know that.  The docs carve out no exception for the
PyInt_AS_LONG
API function, and since that doesn't appear to exist in Python 3 anymore
(the only Python source I have on my box right now) I couldn't stare at the
implementation to guess.  Staring at implementations is the _only_ way to
guess.


> If they want to muck with the `buf` member without the GIL, then if they
> > care about races among multiple threads mucking with `buf`, they'll have
> to
> > supply their own mutual exclusion mechanism.
>


> Define "muck with".


Precisely my point:  _your_ unqualified "safe" is missing all the
qualifications needed to spell out when it's actually safe.  "safe" is a
subtler question than you're painting it to be ;-)

As long as you're reading the memory area(s)

pointed to by the Py_buffer object, you're fine.  If you plan to write
> to that memory, obviously you'll need to make sure the various threads
> don't overwrite each other, i.e. distribute the work carefully.
>
> But that's a normal provision for multi-threaded algorithms, not a
> Python-specific restriction.  I'm assuming someone asking a question
> about multi-threaded access to CPython objects already knows all
> this :-)


While I don't assume that.  Looking it up, if they stared at

#define  PyInt_AS_LONG

(op)   (((PyIntObject

 *)(op))->ob_ival)
in Python 2 and had to _ask_ whether it was safe to invoke without the GIL,
then a more plausible assumption is they're going more by poke-&-hope than
by reasoning.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Const access to CPython objects outside of GIL?

2018-07-17 Thread Radim Řehůřek

Thanks Tim. That's one unambiguous answer.

I did consider posting to python-list, but this seemed somewhat
python-devish. Any appetite for documenting which foundational functions
are const-safe in the sense I described? Perhaps we could help. What kind
of analysis, demand or momentum is needed for the Python dev team to
consider introducing such a contract?

To be honest, I did do some CPython source code staring already. And at
least for the functions we care about, it wasn't all that painful
(PyDict_GetItem being the trickiest). Doing this wholesale might be a
superhuman task, but I'm thinking a practical subset could be relatively
straightforward while still useful, 80/20.



On Tue, Jul 17, 2018 at 7:15 AM, Tim Peters  wrote:

> [Radim Řehůřek ]
>
>> one of our Python projects calls for pretty heavy, low-level
>> optimizations.
>
>
>> We went down the rabbit hole and determined that having access to
>> PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict, unicode)
>> on Python objects **outside of GIL** might be a good-enough solution. The
>> Python objects in question are guaranteed to live and not be mutated
>> externally in any way. They're "frozen" and read-only.
>>
>> Under what conditions is it OK to call these 3 functions on such objects?
>>
>
> From the "initialization, Finalization, and Threads" section of the
> Python/C API Reference Manual:
>
> Therefore, the rule exists that only the thread that has acquired the GIL
>> may operate on Python objects or call Python/C API functions.
>
>
> Under protection of the GIL is the only documented - or intended - way to
> do anything with Python objects, or to call virtually any Python/C API
> function.
>
>
>
>> More generally, what is the CPython 2.7/3.5 contract regarding (lack of)
>> object mutation, and the need for reference counting and synchronization
>> via GIL?
>>
>
> There is no intention to support GIL-free access to any Python objects.
> So that's the contract:  "All warranties are null & void if you do just
> about anything while not holding the GIL".
>
>
>> Which C API functions are safe to call on "const" objects?
>>
>
> None.  If you find some combinations of CPython versions, functions, and
> objects, that happen to work, that's fine, but there's no guarantee they'll
> continue to work, not even across maintenance releases.  Although they'll
> _probably_ continue to work.
>
> Obviously releasing GIL and then calling C API is hacky, but from initial
>> experiments, it seems to work (see https://stackoverflow.com/
>> questions/51351609/can-i-const-access-cpython-objects-without-gil). But
>> I'm wondering if there's a more formal contract around this behaviour.
>>
>
> Nope!  You're on your own here.
>
> Which doesn't mean you can't do it.  Just that if things blow up, you're
> still on your own - since you're far outside what the documentation says is
> required (which I suspect you knew before you asked ;-) ), the project
> won't even accept a bug report.
>
> If you want more help trying to guess which functions _may_ work outside
> the law, that's fine too, but "python-dev" (this mailing list) is for
> development _of_ Python itself, not for questions about _using_ Python.
> The more general python-list would be more appropriate.  But, since you're
> trying to use the C API in unsupported ways it wasn't intended to be used,
> you'll likely have a hard time finding people with significant experience
> doing the same thing.  Since it's not at all an intended use, there are no
> "general principles" at work to ease the pain of staring at the CPython
> source code to try to guess what might go wrong across all code paths.
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Fuzzing the Python standard library

2018-07-17 Thread Jussi Judin

Hi,

I have been fuzzing[1] various parts of Python standard library for Python 3.7 
with python-afl[2] to find out internal implementation issues that exist in the 
library. What I have been looking for are mainly following:

* Exceptions that are something else than the documented ones. These usually 
indicate an internal implementation issue. For example one would not expect an 
UnicodeDecodeError from netrc.netrc() function when the documentation[3] 
promises netrc.NetrcParseError and there is no way to pass properly sanitized 
file object to the netrc.netrc().
* Differences between values returned by C and Python versions of some 
functions. quopri module may have these.
* Unexpected performance and memory allocation issues. These can be somewhat 
controversial to fix, if at all, but at least in some cases from end-user 
perspective it can be really nasty if for example 
fractions.Fraction("1.64E664644") results in hundreds of megabytes of 
memory allocated and takes very long to calculate. I gave up waiting for that 
function call to finish after 5 minutes.

As this is going to result in a decent amount of bug reports (currently I only 
filed one[4], although that audio processing area has much more issues to 
file), I would like to ask your opinion on filing these bug reports. Should I 
report all issues regarding some specific module in one bug report, or try to 
further split them into more fine grained reports that may be related? These 
different types of errors are specifically noticeable in zipfile module that 
includes a lot of different exception and behavioral types on invalid data 
 . 
And in case of sndhdr module, there are multiple modules with issues (aifc, 
sunau, wave) that then show up also in sndhdr when they are used. Or are some 
of you willing to go through the crashes that pop up and help with the report 
filing?

The code and more verbose description for this is available from 
. It works by default on some 
GNU/Linux systems only (I use Debian testing), as it relies on /dev/shm/ being 
available and uses shell scripts as wrappers that rely on various tools that 
may not be installed on all systems by default.

As a bonus, as this uses coverage based fuzzing, it also opens up the 
possibility of automatically creating a regression test suite for each of the 
fuzzed modules to ensure that the existing functionality (input files under 
/corpus/ directory) does not suddenly result in additional 
exceptions and that it is more easy to test potential bug fixes (crash inducing 
files under /crashes/ directory).

As a downside, this uses two quite specific tools (afl, python-afl) that have 
further dependencies (Cython) inside them, I doubt the viability of integrating 
this type of testing as part of normal Python verification process. As a 
difference to libFuzzer based fuzzing that is already integrated in Python[5], 
this instruments the actual (and only the) Python code and not the actions that 
the interpreter does in the background. So this should result in better fuzzer 
coverage for Python code that is used with the downside that when C functions 
are called, they are complete black boxes to the fuzzer.

I have mainly run these fuzzer instances at most for several hours per module 
with 4 instances and stopped running no-issue modules after there have been no 
new coverage discovered after more than 10 minutes. Also I have not really 
created high quality initial input files, so I wouldn't be surprised if there 
are more issues lurking around that could be found with throwing more CPU and 
higher quality fuzzers at the problem.

[1]: https://en.wikipedia.org/wiki/Fuzzing
[2]: https://github.com/jwilk/python-afl
[3]: https://docs.python.org/3/library/netrc.html
[4]: https://bugs.python.org/issue34088
[5]: https://github.com/python/cpython/tree/3.7/Modules/_xxtestfuzz

-- 
Jussi Judin
https://jjudin.iki.fi/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Question about PEP 484

2018-07-17 Thread Sebastian Rittau


On 17.07.2018 17:05, Guido van Rossum wrote:
This is a good point. I presume specifying this unambiguously would be 
a huge amount of work, and it would mostly codify mypy's current 
behavior. I don't think that's within the scope of PEP 484, but it 
could well be done as a separate PEP (perhaps an informational one?). 
I hope you understand that I am not volunteering.
An informational PEP sounds about right to me. Such a PEP could also 
include style recommendations like those from typeshed's CONTRIBUTING 
file (https://github.com/python/typeshed/blob/master/CONTRIBUTING.md).


I guess I just volunteered to help with such a PEP, although I feel that 
someone from mypy's core team should take the lead on that. And if I 
understood this thread correctly, the pytype team is also willing to 
help out?


 - Sebastian

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

On Tue, Jul 17, 2018 at 9:24 AM, Serhiy Storchaka 
wrote:

> 17.07.18 18:48, Guido van Rossum пише:
>
>> On Tue, Jul 17, 2018 at 8:28 AM, Serhiy Storchaka > > wrote:
>> Should not the assert statement introduce an implicit lexical scope
>> for preventing leaking variables?
>>
>> I don't see why. As Chris said, side effects in asserts are nothing new
>> and this PEP is not the one to do something about it.
>>
>
> This side effect is new. No other expressions that can be used in asserts
> leaked local variables before. The only exception is list comprehensions in
> Python 2, and this was fixed in Python 3.
>
> We can't make the assignment expression itself creating its own scope,
> because this will invalidate its purpose. But the problem with assert
> ccould be solved by making assert creating a new lexical scope.
>
> assert expr, msg
>
> could be translated to
>
> if __debug__ and not (lambda: expr)():
> raise AssertionError(msg)
>
> instead of
>
> if __debug__ and not expr:
> raise AssertionError(msg)
>

I don't believe this "problem" needs to be solved.

It seems to be you are seeking an excuse to reopen the acrimonious PEP 572
discussion. Please stop.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fuzzing the Python standard library

2018-07-17 Thread Damian Shaw

I'm not a core Python Dev, but quick question, why would you expect "
fractions.Fraction("1.64E664644")" not to take 100s of megabytes and
hours to run?

Simply evaluating: 164 * 10**66464 will take hundreds of megabytes by
definition.

Regards
Damian


On Tue, Jul 17, 2018, 12:54 Jussi Judin  wrote:

> Hi,
>
> I have been fuzzing[1] various parts of Python standard library for Python
> 3.7 with python-afl[2] to find out internal implementation issues that
> exist in the library. What I have been looking for are mainly following:
>
> * Exceptions that are something else than the documented ones. These
> usually indicate an internal implementation issue. For example one would
> not expect an UnicodeDecodeError from netrc.netrc() function when the
> documentation[3] promises netrc.NetrcParseError and there is no way to pass
> properly sanitized file object to the netrc.netrc().
> * Differences between values returned by C and Python versions of some
> functions. quopri module may have these.
> * Unexpected performance and memory allocation issues. These can be
> somewhat controversial to fix, if at all, but at least in some cases from
> end-user perspective it can be really nasty if for example
> fractions.Fraction("1.64E664644") results in hundreds of megabytes of
> memory allocated and takes very long to calculate. I gave up waiting for
> that function call to finish after 5 minutes.
>
> As this is going to result in a decent amount of bug reports (currently I
> only filed one[4], although that audio processing area has much more issues
> to file), I would like to ask your opinion on filing these bug reports.
> Should I report all issues regarding some specific module in one bug
> report, or try to further split them into more fine grained reports that
> may be related? These different types of errors are specifically noticeable
> in zipfile module that includes a lot of different exception and behavioral
> types on invalid data <
> https://github.com/Barro/python-stdlib-fuzzers/tree/master/zipfile/crashes>
> . And in case of sndhdr module, there are multiple modules with issues
> (aifc, sunau, wave) that then show up also in sndhdr when they are used. Or
> are some of you willing to go through the crashes that pop up and help with
> the report filing?
>
> The code and more verbose description for this is available from <
> https://github.com/Barro/python-stdlib-fuzzers>. It works by default on
> some GNU/Linux systems only (I use Debian testing), as it relies on
> /dev/shm/ being available and uses shell scripts as wrappers that rely on
> various tools that may not be installed on all systems by default.
>
> As a bonus, as this uses coverage based fuzzing, it also opens up the
> possibility of automatically creating a regression test suite for each of
> the fuzzed modules to ensure that the existing functionality (input files
> under /corpus/ directory) does not suddenly result in
> additional exceptions and that it is more easy to test potential bug fixes
> (crash inducing files under /crashes/ directory).
>
> As a downside, this uses two quite specific tools (afl, python-afl) that
> have further dependencies (Cython) inside them, I doubt the viability of
> integrating this type of testing as part of normal Python verification
> process. As a difference to libFuzzer based fuzzing that is already
> integrated in Python[5], this instruments the actual (and only the) Python
> code and not the actions that the interpreter does in the background. So
> this should result in better fuzzer coverage for Python code that is used
> with the downside that when C functions are called, they are complete black
> boxes to the fuzzer.
>
> I have mainly run these fuzzer instances at most for several hours per
> module with 4 instances and stopped running no-issue modules after there
> have been no new coverage discovered after more than 10 minutes. Also I
> have not really created high quality initial input files, so I wouldn't be
> surprised if there are more issues lurking around that could be found with
> throwing more CPU and higher quality fuzzers at the problem.
>
> [1]: https://en.wikipedia.org/wiki/Fuzzing
> [2]: https://github.com/jwilk/python-afl
> [3]: https://docs.python.org/3/library/netrc.html
> [4]: https://bugs.python.org/issue34088
> [5]: https://github.com/python/cpython/tree/3.7/Modules/_xxtestfuzz
>
> --
> Jussi Judin
> https://jjudin.iki.fi/
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/damian.peter.shaw%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Question about PEP 484

2018-07-17 Thread Jelle Zijlstra

2018-07-17 9:55 GMT-07:00 Sebastian Rittau :

> On 17.07.2018 17:05, Guido van Rossum wrote:
>
>> This is a good point. I presume specifying this unambiguously would be a
>> huge amount of work, and it would mostly codify mypy's current behavior. I
>> don't think that's within the scope of PEP 484, but it could well be done
>> as a separate PEP (perhaps an informational one?). I hope you understand
>> that I am not volunteering.
>>
> An informational PEP sounds about right to me. Such a PEP could also
> include style recommendations like those from typeshed's CONTRIBUTING file (
> https://github.com/python/typeshed/blob/master/CONTRIBUTING.md).
>
> I guess I just volunteered to help with such a PEP, although I feel that
> someone from mypy's core team should take the lead on that. And if I
> understood this thread correctly, the pytype team is also willing to help
> out?

I can also help out.

>
>
>  - Sebastian
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jelle.
> zijlstra%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

2018-07-17 Thread Steven D'Aprano

On Tue, Jul 17, 2018 at 07:24:12PM +0300, Serhiy Storchaka wrote:
> 17.07.18 18:48, Guido van Rossum пише:
> >On Tue, Jul 17, 2018 at 8:28 AM, Serhiy Storchaka  >> wrote:
> >Should not the assert statement introduce an implicit lexical scope
> >for preventing leaking variables?
> >
> >I don't see why. As Chris said, side effects in asserts are nothing new 
> >and this PEP is not the one to do something about it.
> 
> This side effect is new. No other expressions that can be used in 
> asserts leaked local variables before.

assert vars().__setitem__('foo', 42) or foo

Not that anyone would write such a thing (and it probably won't work 
inside a function in CPython), but it is possible.

Besides, there is a legitimate use for assignment expressions in 
assertions:

assert (spam := something) > 2, 'not enough spam'
assert  sequence[foo] == 999, 'sequence item isn't 999'

Sometimes you need two assertions.

> We can't make the assignment expression itself creating its own scope, 
> because this will invalidate its purpose. But the problem with assert 
> ccould be solved by making assert creating a new lexical scope.

Assertions could be used to perform imports too:

assert __import__('math')

but people don't do that either. Let's not try fixing "problems" that 
won't exist. Most people have more sense than to do that.

And for those who don't and misuse assertions, well, they've been 
misusing them for two decades and the sky hasn't fallen.

> assert expr, msg
> 
> could be translated to
> 
> if __debug__ and not (lambda: expr)():
> raise AssertionError(msg)

YAGNI.

I do a lot of work in the REPL where I'm not afraid to bend the rules 
and use hacks I'd never use in long-lasting code. I'd never use assert 
for testing input in real code, but I do it in the REPL all the time.

If I choose to abuse assertions in the REPL, you ought to trust me to do 
this responsibly. I'm a consenting adult. If I want to run with 
scissors, point a loaded gun at my foot, put all my eggs in one basket, 
or use assignment expressions in an assertion, I don't want the 
interpreter telling me I can't.

-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Question about PEP 484

2018-07-17 Thread Adam Cataldo via Python-Dev

Hi Sebastian,

Of course, we'd be happy to work with you on this!  We just need to figure
out which of us will drive this on our end (most likely Rebecca or Teddy).
I'll huddle with the team and get back to you with an answer on who later
today.



On Tue, Jul 17, 2018 at 9:58 AM Sebastian Rittau  wrote:

> On 17.07.2018 17:05, Guido van Rossum wrote:
> > This is a good point. I presume specifying this unambiguously would be
> > a huge amount of work, and it would mostly codify mypy's current
> > behavior. I don't think that's within the scope of PEP 484, but it
> > could well be done as a separate PEP (perhaps an informational one?).
> > I hope you understand that I am not volunteering.
> An informational PEP sounds about right to me. Such a PEP could also
> include style recommendations like those from typeshed's CONTRIBUTING
> file (https://github.com/python/typeshed/blob/master/CONTRIBUTING.md).
>
> I guess I just volunteered to help with such a PEP, although I feel that
> someone from mypy's core team should take the lead on that. And if I
> understood this thread correctly, the pytype team is also willing to
> help out?
>
>   - Sebastian
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/acataldo%40google.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Question about PEP 484

2018-07-17 Thread Adam Cataldo via Python-Dev

Great Jelle! We look forward to working with you to.

On Tue, Jul 17, 2018 at 10:41 AM Jelle Zijlstra 
wrote:

>
>
> 2018-07-17 9:55 GMT-07:00 Sebastian Rittau :
>
>> On 17.07.2018 17:05, Guido van Rossum wrote:
>>
>>> This is a good point. I presume specifying this unambiguously would be a
>>> huge amount of work, and it would mostly codify mypy's current behavior. I
>>> don't think that's within the scope of PEP 484, but it could well be done
>>> as a separate PEP (perhaps an informational one?). I hope you understand
>>> that I am not volunteering.
>>>
>> An informational PEP sounds about right to me. Such a PEP could also
>> include style recommendations like those from typeshed's CONTRIBUTING file (
>> https://github.com/python/typeshed/blob/master/CONTRIBUTING.md).
>>
>> I guess I just volunteered to help with such a PEP, although I feel that
>> someone from mypy's core team should take the lead on that. And if I
>> understood this thread correctly, the pytype team is also willing to help
>> out?
>
> I can also help out.
>
>>
>>
>>  - Sebastian
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/jelle.zijlstra%40gmail.com
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/acataldo%40google.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

2018-07-17 Thread Barry Warsaw

On Jul 17, 2018, at 07:59, Guido van Rossum  wrote:
> 
> Personally I like Barry's example just fine -- assuming `subdirs` is not used 
> later, this feels like a good use case.

Thanks!  I thought it was cute.  It was just something that occurred to me as I 
was reviewing some existing code.  The intent wasn’t to use `subdirs` outside 
of the assert statement, but I’m warm to it because it means I don’t have to do 
wasted work outside of the assert statement, or repeat myself in the assert 
message part.

-Barry

signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

[Barry Warsaw]

> Thanks!  I thought it was cute.  It was just something that occurred to me
> as I was reviewing some existing code.  The intent wasn’t to use `subdirs`
> outside of the assert statement, but I’m warm to it because it means I
> don’t have to do wasted work outside of the assert statement, or repeat
> myself in the assert message part.
>

Because the latter ("repeat myself") is probably more tempting, I'll just
note that the "laziness" of using an assignment expression instead may well
have nudged you toward writing _better_ code too.

   assert len(subdirs := list(path.iterdir())) == 0, subdirs

Assuming the result of list(path.iterdir()) can change over time (seems
very likely),

   assert len(list(path.iterdir())) == 0,  list(path.iterdir())

_could_ end up both triggering and displaying an empty list in the
exception detail.  The assignment-expression version cannot.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

2018-07-17 Thread Barry Warsaw

On Jul 17, 2018, at 11:34, Tim Peters  wrote:

> Assuming the result of list(path.iterdir()) can change over time (seems very 
> likely),
> 
>assert len(list(path.iterdir())) == 0,  list(path.iterdir())
> 
> _could_ end up both triggering and displaying an empty list in the exception 
> detail.  The assignment-expression version cannot.

Tim, you’re dangerously tempting me into +1 territory. :)

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

2018-07-17 Thread MRAB


On 2018-07-17 19:34, Tim Peters wrote:


[Barry Warsaw]

Thanks!  I thought it was cute.  It was just something that occurred
to me as I was reviewing some existing code.  The intent wasn’t to
use `subdirs` outside of the assert statement, but I’m warm to it
because it means I don’t have to do wasted work outside of the
assert statement, or repeat myself in the assert message part.


Because the latter ("repeat myself") is probably more tempting, I'll 
just note that the "laziness" of using an assignment expression instead 
may well have nudged you toward writing _better_ code too.


    assert len(subdirs := list(path.iterdir())) == 0, subdirs

Assuming the result of list(path.iterdir()) can change over time (seems 
very likely),


    assert len(list(path.iterdir())) == 0, list(path.iterdir())

_could_ end up both triggering and displaying an empty list in the 
exception detail.  The assignment-expression version cannot.



Why use len(...) == 0 instead of not(...)?

assert not(subdirs := list(path.iterdir())), subdirs
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572 and assert

2018-07-17 Thread Barry Warsaw

On Jul 17, 2018, at 12:18, MRAB  wrote:

> Why use len(...) == 0 instead of not(...)?
> 
>assert not(subdirs := list(path.iterdir())), subdirs

Off-topic, but it’s a style thing.  Some style guides (and my personal 
preference) require to be more explicit when checking for empty sequences.  
I’ll use a boolean test (e.g. `not` or false-iness) when the subject could be 
empty, None, or the empty string, and `len() == 0` when the subject is supposed 
to be a sequence.

-Barry

signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Const access to CPython objects outside of GIL?

[Radim Řehůřek ]

> Thanks Tim. That's one unambiguous answer.


I'm glad that part came across ;-)


> I did consider posting to python-list, but this seemed somewhat
> python-devish.
>

I agree it's on the margin - I don't mind.

Any appetite for documenting which foundational functions are const-safe in
> the sense I described? Perhaps we could help. What kind of analysis,
> demand or momentum is needed for the Python dev team to consider
> introducing such a contract?
>

Most are volunteers scratching their own itches, and sometimes paid to
scratch their employers' itches.  This isn't my itch, so not me.  Perhaps
someone else - but from the messages so far it appears that most who
replied suspect this is an "XY problem":

https://en.wikipedia.org/wiki/XY_problem

Python's been around for decades, and best I recall nobody has suggested
anything really akin to what you're asking for here.  There certainly are C
extensions that want GIL-free access to (possibly massive amounts of) data
visible from Python too.  `numpy` is the prime example of that.  As Antoine
hinted, people working on those came up with the "buffer protocol" instead
(documented in the Python/C API Reference Manual under the "Buffer
Protocol" section).

But nobody here knows anything about your app, so can't guess from here
whether the buffer protocol is of any interest to you.  The buffer protocol
is aimed at uniform layouts of (possibly massive amounts of) native
C-format data, not at GIL-free access to individual little Python objects.

To be honest, I did do some CPython source code staring already.


Which is the only way you can learn anything about whether breaking
fundamental rules can get you in trouble.


> And at least for the functions we care about, it wasn't all that painful
> (PyDict_GetItem being the trickiest). Doing this wholesale might be a
> superhuman task, but I'm thinking a practical subset could be relatively
> straightforward while still useful, 80/20.
>

If you want to pursue this, it's probably necessary to first specify the
set of API functions that need to change their requirements and promises,
and to specify exactly what they're supposed to require and promise instead.

Even then, you should expect resistance.  It's been, historically, hard
enough to avoid thread bugs in CPython's implementation even with the
blanket "no access without the GIL, period" current requirement.  Adding
some number of "but these functions don't care about the GIL, in some
circumstances they can't verify obtain, let alone enforce" exceptions would
complicate an area that's already delicate.

CPython's implementation not only requires that only one thread holds the
GIL, it also works to verity that at runtime, and raises a fatal exception
if it detects that it's even slightly confused about which thread currently
holds the GIL.  Adding weaker but unverifiable preconditions to some
functions isn't attractive on the face of it.

Note:  the kind of people who find the GIL extremely intrusive tend instead
to work on ways to eliminate the GIL entirely.  They typically give up
after a few years of intense pain ;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2to3 for python annotations

2018-07-17 Thread Philippe Fremy

Hi,

While contributing to pyannotate, I became familiar enough with 2to3
fixers to be able to convert Python 2 style annotations to Python 3.

Is it something that would be interesting to put into python 2to3 ? If
so I would propose a PR for this.

Cheers,

Philippe

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2to3 for python annotations

2018-07-17 Thread Jelle Zijlstra

2018-07-17 12:37 GMT-07:00 Philippe Fremy :

> Hi,
>
> While contributing to pyannotate, I became familiar enough with 2to3
> fixers to be able to convert Python 2 style annotations to Python 3.
>
> Is it something that would be interesting to put into python 2to3 ? If
> so I would propose a PR for this.
>
> There already a tool that does this:
https://github.com/ilevkivskyi/com2ann.

I am not sure it would be appropriate for 2to3. Type comments still work in
Python 3, and I feel like 2to3 should do the minimum necessary to get
working code in Python 3.


> Cheers,
>
> Philippe
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> jelle.zijlstra%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2to3 for python annotations