Re: [Python-ideas] Importance of noticing new signals
On Fri, 3 Nov 2017 17:22:07 +0200 Koos Zevenhoven wrote: > > Maybe you are concerned about whether some nuances and recent changes to > signal handling could lead to harmful change in behavior in some meaningful > edge cases? I can at least say that my PyErr_PROBE_SIGNALS() proposal does > not introduce such issues, if the difference is documented properly: > > """PyErr_PROBE_SIGNALS() is meant for performance-critical code and is not > 100% guaranteed to always see the most recent signals. If a signal being > deferred is a concern, use PyErr_CheckSignals() instead.""" Agreed. > But more generally, if we could assume that trip_signal() and > PyErr_CheckSignals() always happen in the same "CPU thread", then we > wouldn't need pyatomic.h here at all. The fact that the code currently > assumes that all Python signal handlers should run in the same Python > thread takes care of some of these concerns without needing locks etc. Hmm... *Python* signal handlers (registered with signal.signal()) all run in the same thread, but we have no way to ensure to ensure that trip_signal() (which is a C-level signal handler called by the OS) will run in the same thread. Even if we were to mask all signals in non-main threads started by Python (e.g. with threading.Thread), there can still be threads created by third-party C libraries that we don't have any control over. > Some other concerns I can imagine by looking at some of the code in > Modules/signalmodule.c: > > (1) If trip_signal() and PyErr_CheckSignals() are executed concurrently, > trip_signal() might set a new signal flag (using relaxed memory order) > while PyErr_CheckSignals is still running. Then if PyErr_CheckSignals() > sets is_tripped to zero *after* trip_signal() sets it to 1, then the new > signal might be deferred until the next time *some* new signal arrives, > which could take an arbitrarily long amount of time, I suppose. > > However, it looks like this problem has been solved by always setting > is_tripped to zero (with strict SEQ_CST memory order) *before* handling the > individual signals. So if trip_signal() has already set is_tripped to 1 > (with SEQ_CST), that prevents PyErr_CheckSignals from setting is_tripped to > zero (with SEQ_CST) *and not* handling the signal. If trip_signal() has not > yet finished, and therefore not set is_tripped to 1 yet, it will cause the > next call to PyErr_CheckSignals to catch the signal. > > (2) Again, if trip_signal() and PyErr_CheckSignals() execute concurrently, > it might happen that PyErr_CheckSignals() handles the signal *before* > trip_signal sets is_tripped to 1. That would cause the next call to > PyErr_CheckSignals() to think there's an unhandled signal, but will most > likely not find one, because it was already handled on the previous call. > But that just effectively means that nothing is done. In fact, there's a > comment in the code that mentions this. > > (3, 4, ...) Of course there's more to take care of there, but that's > unrelated to my PyErr_PROBE_SIGNALS() proposal. Anyway, at least (1) and > (2) seem to already have been taken care of, and I assume you are aware of > that. Yes, that was my analysis too (and why I implemented it this way in the first place). But I'm glad you confirm it, since on such topics it's always easy to miss a potential problem. Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Importance of noticing new signals
On Thu, Nov 2, 2017 at 2:57 PM, Koos Zevenhoven wrote:
> On Thu, Nov 2, 2017 at 2:22 PM, Antoine Pitrou
> wrote:
>
>> On Wed, 1 Nov 2017 20:29:56 +0200
>> Koos Zevenhoven wrote:
>> >
>> > From a correctness point of view, that is absolutely great: if
>> > PyErr_CheckSignals() is called, it is guaranteed to notice a new signal
>> > regardles of how small the number of picoseconds after the `is_tripped`
>> > flag has been set. But is that really important?
>>
>> I was going to answer "no"... but actually yes. The important case is
>> the event loop type case:
>>
>> while (1) {
>> select([some file descriptors]);
>> if (errno == EINTR) {
>> PyErr_CheckSignals();
>> if (PyErr_Occurred()) break;
>> }
>> /* continue select()ing... */
>> }
>>
>> Now say at a given point in time, no fds are actually active (or even
>> waited for), but some signal arrives (SIGINT perhaps).
>> select() is woken up and returns with errno EINTR. Two things then can
>> happen:
>>
>> - if PyErr_CheckSignals() notices the signal, it will run the relevant
>> signal handler, which may raise an exception and trigger the select()
>> loop to exit (e.g. SIGINT would raise KeyboardInterrupt)
>>
>> - if PyErr_CheckSignals() misses the signal, the loop will enter again,
>> and select() may sleep for an infinite amount of time
>>
>>
> Oh! So that would provide a proper reason for my just-in-case decision to
> name the faster near-equivalent functionality PyErr_PROBE_SIGNALS instead
> of PyErr_CHECK_SIGNALS.
>
> Cross-referencing to that (thread about making Ctrl-C "always" work):
>
> https://mail.python.org/pipermail/python-ideas/2017-November/047631.html
>
>
> Of course, what we're doing with select() above can already apply for
>> read() or other interruptible syscalls waiting for outside data... and
>> that pattern is present a lot internally, especially since
>>
>> PEP 475 ("Retry system calls failing with EINTR").
>>
>
>> Now, is the "sequentially consistent" ordering on is_tripped sufficient
>> to guarantee that signals won't be missed on a weak-ordering platform?
>> I *think* so, but an expert would need to check that code (or we
>> cross our fingers and wait for a hypothetical bug report).
>>
>>
> I think the question is: Do we know for sure that is_tripped has been
> stored using sequentially consistent ordering prior to the call to
> PyErr_CheckSignals(), even if an interruptible syscall is involved? I
> suppose so?
>
> (But this is a separate question from the problem I was solving, of
> course. I'm not proposing to remove PyErr_CheckSignals())
>
>
To continue on this: If I understand your question correctly, I'm hesitant
to make strong statements about it. It would be interesting to know what we
can assume about signals that happen at the same time with system calls,
given the various platforms supported. Unfortunately, I don't know that.
Maybe you are concerned about whether some nuances and recent changes to
signal handling could lead to harmful change in behavior in some meaningful
edge cases? I can at least say that my PyErr_PROBE_SIGNALS() proposal does
not introduce such issues, if the difference is documented properly:
"""PyErr_PROBE_SIGNALS() is meant for performance-critical code and is not
100% guaranteed to always see the most recent signals. If a signal being
deferred is a concern, use PyErr_CheckSignals() instead."""
But more generally, if we could assume that trip_signal() and
PyErr_CheckSignals() always happen in the same "CPU thread", then we
wouldn't need pyatomic.h here at all. The fact that the code currently
assumes that all Python signal handlers should run in the same Python
thread takes care of some of these concerns without needing locks etc.
Some other concerns I can imagine by looking at some of the code in
Modules/signalmodule.c:
(1) If trip_signal() and PyErr_CheckSignals() are executed concurrently,
trip_signal() might set a new signal flag (using relaxed memory order)
while PyErr_CheckSignals is still running. Then if PyErr_CheckSignals()
sets is_tripped to zero *after* trip_signal() sets it to 1, then the new
signal might be deferred until the next time *some* new signal arrives,
which could take an arbitrarily long amount of time, I suppose.
However, it looks like this problem has been solved by always setting
is_tripped to zero (with strict SEQ_CST memory order) *before* handling the
individual signals. So if trip_signal() has already set is_tripped to 1
(with SEQ_CST), that prevents PyErr_CheckSignals from setting is_tripped to
zero (with SEQ_CST) *and not* handling the signal. If trip_signal() has not
yet finished, and therefore not set is_tripped to 1 yet, it will cause the
next call to PyErr_CheckSignals to catch the signal.
(2) Again, if trip_signal() and PyErr_CheckSignals() execute concurrently,
it might happen that PyErr_CheckSignals() handles the signal *before*
trip_signal sets is_tripped to 1. That would
Re: [Python-ideas] Moving typing out of the stdlib in Python 3.7?
On Fri, 3 Nov 2017 16:04:43 +0100 Stéfane Fermigier wrote: > I use typing quite systematically nowadays, even for projects that don't > use mypy (for historical reasons: bringing an older code base to zero mypy > issues can be quite time-consuming). > > For instance, adding typing annotation can help autocompletion under > PyCharm (and hopefully other IDEs). > > With these annotations, PyCharm is also able to signal typing issues either > directly in the editor, or when running a code check. > > I'm quite OK with removing the typing module from the stdlib as it can > easily be added to my projects dependencies, and I can definitively > understand the benefits of a faster release cycle, but I'm worried that > this could hinder adoption of these practices by certain people. I don't think casual or beginner users of Python really have to worry about typing annotations. As I understand it, they become really useful on middle- to large-scale projects (disclaimer: I've never used them myself; the kind of typing the Numba project does -- which I don't participate in anymore -- is quite different). Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Moving typing out of the stdlib in Python 3.7?
I use typing quite systematically nowadays, even for projects that don't use mypy (for historical reasons: bringing an older code base to zero mypy issues can be quite time-consuming). For instance, adding typing annotation can help autocompletion under PyCharm (and hopefully other IDEs). With these annotations, PyCharm is also able to signal typing issues either directly in the editor, or when running a code check. I'm quite OK with removing the typing module from the stdlib as it can easily be added to my projects dependencies, and I can definitively understand the benefits of a faster release cycle, but I'm worried that this could hinder adoption of these practices by certain people. S. On Fri, Nov 3, 2017 at 3:35 PM, Guido van Rossum wrote: > [A copy from https://github.com/python/typing/issues/495 to get more > people's attention to this issue.] > > I'm wondering if we should remove typing from the stdlib. Now's the time > to think about this, as the feature freeze for 3.7 is about 12 weeks away. > > Cons: > >- People have to depend on a PyPI package to use typing (but they do >anyway for typing_extensions) >- It's a backward incompatibility for users of Python 3.5 and 3.6 (but >the typing module was always provisional) > > Pros: > >- The typing module can evolve much faster outside the stdlib >- We could get rid of typing_extensions (and maybe even mypy_extensions >) > > If we don't do this I worry that we're entering a period where many new > typesystem features end up in typing_extensions and users will be > confused about which items are in typing and which in typing_extensions > (not to mention mypy_extensions). Anything new to be added to typing > (e.g. Const, Final, Literal, or changing ABCs to Protocols) would have to > be added to typing_extensions instead, and users would be confused about > which features exist in which module. Moving typing out of the stdlib can > make things potentially simpler, at the cost of an extra pip install (but > they'll need one anyway for mypy). > > Thoughts? > > -- > --Guido van Rossum (python.org/~guido) > > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- “You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.” — R. Buckminster Fuller ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Moving typing out of the stdlib in Python 3.7?
On Sat, Nov 4, 2017 at 1:49 AM, Guido van Rossum wrote: > On Fri, Nov 3, 2017 at 7:45 AM, Chris Angelico wrote: >> >> On Sat, Nov 4, 2017 at 1:35 AM, Guido van Rossum wrote: >> > [A copy from https://github.com/python/typing/issues/495 to get more >> > people's attention to this issue.] >> > >> > I'm wondering if we should remove typing from the stdlib. Now's the time >> > to >> > think about this, as the feature freeze for 3.7 is about 12 weeks away. >> > >> > Cons: >> > >> > People have to depend on a PyPI package to use typing (but they do >> > anyway >> > for typing_extensions) >> >> If the lazy evaluation of annotations (PEP 563) also lands in 3.7, >> then this would be a very minor downside. You'd need to pip-install >> typing as well as mypy *for the actual type checking*, but at run >> time, you could ignore both (all those List[...] annotations would be >> stored unevaluated). Otherwise, it'd mean that any project that makes >> use of type hints would require typing as a run-time dependency. > > > This would not work if you use TypeVar, NewType, or any kind of type alias > involving things imported from typing (e.g. Union or TypedDict). Also > cast(). Ah, I forgot about those - they're not just used in the annotations. Oh well, was a nice idea. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Moving typing out of the stdlib in Python 3.7?
On Fri, Nov 3, 2017 at 7:45 AM, Chris Angelico wrote: > On Sat, Nov 4, 2017 at 1:35 AM, Guido van Rossum wrote: > > [A copy from https://github.com/python/typing/issues/495 to get more > > people's attention to this issue.] > > > > I'm wondering if we should remove typing from the stdlib. Now's the time > to > > think about this, as the feature freeze for 3.7 is about 12 weeks away. > > > > Cons: > > > > People have to depend on a PyPI package to use typing (but they do anyway > > for typing_extensions) > > If the lazy evaluation of annotations (PEP 563) also lands in 3.7, > then this would be a very minor downside. You'd need to pip-install > typing as well as mypy *for the actual type checking*, but at run > time, you could ignore both (all those List[...] annotations would be > stored unevaluated). Otherwise, it'd mean that any project that makes > use of type hints would require typing as a run-time dependency. > This would not work if you use TypeVar, NewType, or any kind of type alias involving things imported from typing (e.g. Union or TypedDict). Also cast(). -- --Guido van Rossum (python.org/~guido) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Moving typing out of the stdlib in Python 3.7?
On Sat, Nov 4, 2017 at 1:35 AM, Guido van Rossum wrote: > [A copy from https://github.com/python/typing/issues/495 to get more > people's attention to this issue.] > > I'm wondering if we should remove typing from the stdlib. Now's the time to > think about this, as the feature freeze for 3.7 is about 12 weeks away. > > Cons: > > People have to depend on a PyPI package to use typing (but they do anyway > for typing_extensions) If the lazy evaluation of annotations (PEP 563) also lands in 3.7, then this would be a very minor downside. You'd need to pip-install typing as well as mypy *for the actual type checking*, but at run time, you could ignore both (all those List[...] annotations would be stored unevaluated). Otherwise, it'd mean that any project that makes use of type hints would require typing as a run-time dependency. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Moving typing out of the stdlib in Python 3.7?
[A copy from https://github.com/python/typing/issues/495 to get more people's attention to this issue.] I'm wondering if we should remove typing from the stdlib. Now's the time to think about this, as the feature freeze for 3.7 is about 12 weeks away. Cons: - People have to depend on a PyPI package to use typing (but they do anyway for typing_extensions) - It's a backward incompatibility for users of Python 3.5 and 3.6 (but the typing module was always provisional) Pros: - The typing module can evolve much faster outside the stdlib - We could get rid of typing_extensions (and maybe even mypy_extensions) If we don't do this I worry that we're entering a period where many new typesystem features end up in typing_extensions and users will be confused about which items are in typing and which in typing_extensions (not to mention mypy_extensions). Anything new to be added to typing (e.g. Const, Final, Literal, or changing ABCs to Protocols) would have to be added to typing_extensions instead, and users would be confused about which features exist in which module. Moving typing out of the stdlib can make things potentially simpler, at the cost of an extra pip install (but they'll need one anyway for mypy). Thoughts? -- --Guido van Rossum (python.org/~guido) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] install pip packages from Python prompt
> > Reading this I get the impression, and correct me if I'm wrong, that > you've never taught beginners programming. Of course long term (heck in > fact fairly early on) they need to learn these nitty-gritty and > sometimes frustrating lessons, but not in a 2 hour intro to programming > for total beginners. > > And I beg to differ--this field is for everyone, and increasingly moreso > every day. Doesn't mean it's easy, but it is and can be for everyone. > > Whether this specific proposal is technically feasible in a > cross-platform manner with the state of the Python interpreter and > import system is another question. But that's a discussion worth having. > "Some people aren't cut out for programming" isn't. > > When teaching an absolute beginners course, (which I personally have done many times often for people with little or no experience of computers), on of the best approaches that I have found is to run JupyterHub on a local (WiFi) LAN - this has several advantages: - the students can get started on some python programming without having to install anything on their own machines - they just need a browser. - No internet connection needed, (often the case in a corporate environment). - I have complete control over the packages present. - Once they have a taste they are ready, usually keen, to learn how to install python and it's libraries, including Jupyter. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
