[Python-Dev] IDLE internal layout

[Louie Lu]

(Due to idle-dev mailing list low traffic, this mail was sent to
python-dev and idle-dev)

Hi all,

I'm now trying to figure out IDLE internal relation.
I got some module dependency graph and count.

The work is at HackMD:

https://hackmd.io/IwNg7AphDGBmsFoAsBOCAmZAGMBDBKsArEgruuiEkQCa5YBGsNQA

Is there any previous material on this topic?


thanks

Louie.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Nick Coghlan]

On 13 June 2017 at 19:48, Victor Stinner  wrote:
> While I'm not opposed to PYTHONCOERCECLOCALE=warn, I'm not sure that
> *developers* will use it. Usually, developers don't care of Unicode
> until enough users complain that their application don't work on a
> specific configuration on a specific platform :-) So I'm not sure that
> it's useful.

It isn't cross-platform app developers that I expect to find it
useful, it's system integrators and operating system developers.
They're users too (and those categories are largely a better
description of my own Python development background than user facing
app development would be).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Nick Coghlan]

On 13 June 2017 at 19:07, Victor Stinner  wrote:
> Ok, if you want to have a more concrete example of regression
> introduced by this warning, look at test_tracemalloc failures on this
> FreeBSD buildbot:
> http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Debug%203.x/builds/419/steps/test/logs/stdio
>
> I expect that a *lot* of tests using stderr will be broken by this warning.
>
> I'm not interested to discuss if the CI is properly configured or not.
> The thing is the warning can easily be seen as a regression: the test
> pass on 3.6 on the same platform ;-)

Those failures aren't due to the successful coercion warning, they're
due to the "Python 3.7 has dropped support for your ASCII-only
platform" warning that the interpreter ends up emitting after coercion
fails :)

You're already working on the more user-friendly solution to that
case, by way of PEP 540 (and if we decide it's a more expedient
interim solution, I'd be fine with switching the Py_WARN_ON_C_LOCALE
config setting to "no" until you've had more time to work on that).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Victor Stinner]

2017-06-13 8:51 GMT+02:00 Nick Coghlan :
> If we turned the warning off by default, but retained an easy
> "on-switch" for the warning to help track places where locale coercion
> is triggering unexpectedly (e.g. "PYTHONCOERCECLOCALE=warn"), that
> would probably give us the best of both worlds.

Hum, it reminds me when I proposed to add a new "locale" encoding
which would be used to explicitly use the locale encoding. My intent
was to emit a warning when the locale encoding was used implicitely...
The problem is that the locale encoding is basically used *everywhere*
in the standard library and *it's fine* :-)

Really, using the locale encoding is the right encoding is many use
cases. Especially when you work on Windows!

Ok, when I proposed the idea, distutils still needed to use UTF-8
explicitly instead of the implicit locale encoding, but you may know
how painful it was to change anything in distutils: write a PEP and/or
discuss it on distutils-sig, handle backward compatibility, etc. The
good news is that these issues are now fixed :-)

--

While I'm not opposed to PYTHONCOERCECLOCALE=warn, I'm not sure that
*developers* will use it. Usually, developers don't care of Unicode
until enough users complain that their application don't work on a
specific configuration on a specific platform :-) So I'm not sure that
it's useful.

We already have many PYTHON* environment variables. I'm not sure that
all of them are used. We should be careful when adding new ones ;-)
https://docs.python.org/dev/using/cmdline.html#environment-variables

It's already possible to implement manually a check in an application
to complain if the LC_CTYPE locale "C", no?

haypo@selma$ cat > x.py
import locale
if locale.setlocale(locale.LC_CTYPE, None) == 'C':
  print("Don't use the C locale")

haypo@selma$ LANG= python2 x.py
Don't use the C locale

haypo@selma$ LANG= python3 x.py
Don't use the C locale

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Victor Stinner]

Ok, if you want to have a more concrete example of regression
introduced by this warning, look at test_tracemalloc failures on this
FreeBSD buildbot:
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Debug%203.x/builds/419/steps/test/logs/stdio

I expect that a *lot* of tests using stderr will be broken by this warning.

I'm not interested to discuss if the CI is properly configured or not.
The thing is the warning can easily be seen as a regression: the test
pass on 3.6 on the same platform ;-)

One example:

==
FAIL: test_conflicting_envvar_and_command_line
(test.test_warnings.CEnvironmentVariableTests)
--
Traceback (most recent call last):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_warnings/__init__.py",
line 1062, in test_conflicting_envvar_and_command_line
b"DeprecationWarning: Message"])
AssertionError: Lists differ: [b'Python runtime initialized with
LC_CTYPE=[305 chars]age'] != [b'Traceback (most recent call last):',
b'  [65 chars]age']

First differing element 0:
b'Python runtime initialized with LC_CTYPE=[190 chars]ded.'
b'Traceback (most recent call last):'

First list contains 1 additional elements.
First extra element 3:
b'DeprecationWarning: Message'

- [b'Python runtime initialized with LC_CTYPE=C (a locale with default
ASCII enco'
-  b'ding), which may cause Unicode compatibility problems. Using
C.UTF-8, C.utf8'
-  b', or UTF-8 (if available) as alternative Unicode-compatible
locales is recom'
-  b'mended.',
-  b'Traceback (most recent call last):',
? ^

+ [b'Traceback (most recent call last):',
? ^

   b'  File "", line 1, in ',
   b'DeprecationWarning: Message']

Victor


2017-06-12 10:56 GMT+02:00 Victor Stinner :
> Hi,
>
> Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step
> forward to UTF-8 everywhere ;-)
>
> I would prefer to not be annoyed by warning messages about encodings
> at startup if possible:
>
> "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
> locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion
> behavior)."
>
>
> Python 3.7 is the only programming language that I know that complains
> about encoding at startup. Even if my code is 100% valid, PEP 8
> compliant, don't emit any warning, etc. I will get the warning. It is
> *not* possible to ignore the warning... Like "yeah, I know that my
> locale is C, but it's not like I'm able to configure it."
>
> haypo@selma$ export LANG=
> haypo@selma$ locale
> LANG=
> LC_CTYPE="POSIX"
> ...
>
> haypo@selma$ ./python -c 'print("Hello World!")' # Python 3.7
> Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
> locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion
> behavior).
> Hello World!
>
> haypo@selma$ python2 -c 'print("Hello World!")' # Python 2.7
> Hello World!
>
> haypo@selma$ perl -e 'print "Hello, world!\n"' # Perl 5.24
> Hello, world!
>
> haypo@selma$ ./c   # C
> Hello World!
>
> ...
>
> Note: I don't consider that 2>/dev/null is a good practice to ignore a
> single warning, since it will ignore *all* message written into
> stderr, including useful warnings like ResourceWarning or "Exception
> ignored ...".
>
> Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handle errors in cleanup code

[Nathaniel Smith]

On Tue, Jun 13, 2017 at 12:10 AM, Nick Coghlan  wrote:

>
> reporting failures from concurrent.futures.wait:
> https://pythonhosted.org/futures/#concurrent.futures.wait


Yeah, and asyncio.gather is another example in the stdlib. Or there's
twisted's DeferredList. Trio is unusual in effectively forcing all tasks to
be run under a gather(), but the basic concept isn't unique at all.


> Figuring out how to *display* an exception tree coherently is going to
> be a pain (it's already problematic with just the linked list), but if
> we can at least model exception trees consistently, then we'd be able
> to share that display logic, even if the scenarios resulting in
> MultiErrors varied.
>

It's true, it was a pain :-). And I'm sure it can be refined. But I think
this is a reasonable first cut:
https://github.com/python-trio/trio/blob/9e0df6159e55fe5e389ae5e24f9bbe51e9b77943/trio/_core/_multierror.py#L341-L388

Basically the algorithm there is:

if there's a __cause__ or __context__:
  print it (recursively using this algorithm)
  print the description line ("this exception was the direct cause" or
whatever is appropriate)
print the traceback and str() attached to this exception itself
for each embedded exception:
  print "Details of embedded exception {i}:"
  with extra indentation:
 print the embedded exception (recursively using this algorithm)
(+ some memoization to avoid infinite recursion on loopy structures)

Of course really complicated trainwrecks that send exception shrapnel
flying everywhere can still be complicated to read, but ... that's why we
make the big bucks, I guess. (My fingers initially typoed that as "that's
why we make the big bugs". Just throwing that out there.)

Currently that code has a hard-coded assumption that the only kind of
exception-container is MultiError, but I guess it wouldn't be too hard to
extend that into some kind of protocol that your asymmetric CleanupError
could also participate in? Like: an abstract exception container has 0 or 1
(predecessor exception, description) pairs [i.e., __cause__ or
__context__], plus 0 or more (embedded exception, description) pairs, and
then the printing code just walks the resulting tree using the above
algorithm?

If this all works out at the library level, then one can start to imagine
ways that the interpreter could potentially get benefits from participating:

- right now, if an exception that already has a __cause__ or __context__ is
re-raised, then implicit exception chaining will overwrite the old
__cause__ or __context__. Instead, it could check to see if there's already
a non-None value there, and if so do exc.__context__ =
MultiError([exc.__context__, new_exc]).

- standardizing the protocol for walking over an exception container would
let us avoid fighting over who owns sys.excepthook, and make it easier for
third-party exception printers (like IPython, pytest, etc.) to play along.

- MultiError.catch is a bit awkward. If we were going all-in on this then
one can imagine some construct like

multitry:
... do stuff ...
raise MultiError([ValueError(1), MultiError([TypeError(),
ValueError(2)])])
multiexcept ValueError as exc:
print("caught a ValueError", exc)
multiexcept TypeError:
print("caught a TypeError and raising RuntimeError")
raise RuntimeError

which prints

  caught a ValueError 1
  caught a TypeError and raising RuntimeError
  caught a ValueError 2

and then raises a RuntimeError with its __context__ pointing to the
TypeError.

...but of course that's pretty speculative at this point; definitely
more python-ideas territory.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handle errors in cleanup code

[Stefan Ring]

On Tue, Jun 13, 2017 at 2:26 AM, Nathaniel Smith  wrote:
> On Mon, Jun 12, 2017 at 6:29 AM, Stefan Ring  wrote:
>>
>> > Yury in the comment for PR 2108 [1] suggested more complicated code:
>> >
>> > do_something()
>> > try:
>> > do_something_other()
>> > except BaseException as ex:
>> > try:
>> > undo_something()
>> > finally:
>> > raise ex
>>
>> And this is still bad, because it loses the back trace. The way we do it is:
>>
>> do_something()
>> try:
>> do_something_other()
>> except BaseException as ex:
>> tb = sys.exc_info()[2]
>> try:
>> undo_something()
>> finally:
>> raise ex, None, tb
>
> Are you testing on python 2? On Python 3 just plain 'raise ex' seems
> to give a sensible traceback for me...

Yes, on Python 2.

Interesting to know that this has changed in Python 3. I'll check this
out immediately.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handle errors in cleanup code

[Nick Coghlan]

On 13 June 2017 at 14:10, Nathaniel Smith  wrote:
> On Mon, Jun 12, 2017 at 1:07 AM, Nick Coghlan  wrote:
>> Since I don't see anything in the discussion so far that *requires*
>> changes to the standard library (aside from "we may want to use this
>> ourselves"), the right place to thrash out the design details is so
>> probably contextlib2: https://github.com/jazzband/contextlib2
>>
>> That's where contextlib.ExitStack was born, and I prefer using it to
>> iterate on context management design concepts, since we can push
>> updates out faster, and if we make bad choices anywhere along the way,
>> they can just sit around in contextlib2, rather than polluting the
>> standard library indefinitely.
>
> I'd also be open to extracting MultiError into a standalone library
> that trio and contextlib2 both consume, if there was interest in going
> that way.

I think that would make sense, as it occurred to me while reading your
post that a construct like MultiError may also be useful when
reporting failures from concurrent.futures.wait:
https://pythonhosted.org/futures/#concurrent.futures.wait

At the moment, it isn't at all clear how best to throw an exception
that reports *all* of the raised exceptions in a concurrent map call,
rather than just the first failure.

So the recurring element we have that's common across all 3 scenarios
is the notion of "peer exceptions", where we're unwinding the stack
for multiple reasons, and we want to keep track of all of them.

The "failed resource cleanup" case then becomes an asymmetric variant
of that, where there's one primary exception (the one that triggered
the cleanup), and one or more secondary failures. Given MultiError,
that could be modelled as a two-tiered tree:

class CleanupError(MultiError): pass

CleanupError([
original_exc,
MultiError([*cleanup_errors]),
])

And if there was no original exception and the resource cleanup failed
unprovoked, you'd indicated that by having just a single child rather
than two:

CleanupError([
MultiError([*cleanup_errors]),
])

Figuring out how to *display* an exception tree coherently is going to
be a pain (it's already problematic with just the linked list), but if
we can at least model exception trees consistently, then we'd be able
to share that display logic, even if the scenarios resulting in
MultiErrors varied.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Nick Coghlan]

On 13 June 2017 at 06:58, Gregory P. Smith  wrote:
> I guess so. It is a system environment configuration problem. But I'd still
> imagine that end users never touch their LANG or LC_* environment variables
> themselves.  (the only time I touch anything is to explicitly set LANG=C on
> occasion when I want text process shell tools to run a lot faster; but I
> hardly count as I live in ascii-land, aka the US)
>
> I'm inclined to suggest it should be silent, but I'm curious if we can get
> some linux distros testing with it enabled during the beta period to see how
> much impact it actually has on users in the real world. We could be talking
> about a minor non-issue.

It's at least affecting Fedora's build system (where we've backported
PEP 538 to the system Python in F26):
https://lists.fedorahosted.org/archives/list/python-de...@lists.fedoraproject.org/thread/VSEGOF76XMBJOAO4C2MORNK3I2VIPOTU/

However, Fedora python-devel's current view on that is that it's a
sign that the build system's configuration is outdated, and we should
be setting the locale to C.UTF-8 instead. (Ditto for the Fedora Docker
images, which are currently expecting systemd to sort out the default
locale settings, so it ends up being left as the C default when
there's no init system running inside the container).

So I think the most compelling argument in favour of making the
warning silent by default is Victor's one: so that "LANG=C python"
"just works", rather than attempting to persuade end users to change
what they've become accustomed to typing in order to get software to
emit otherwise internationalised text in English.

After all, one of the long term goals of PEP 538 is to get to the
point where at least the glibc developers, and hopefully other C
standard library implementation developers, are comfortable having
UTF-8 as the default text encoding anyway, at which point CPython's
locale coercion would essentially become a no-op (and folks would have
to type something like "LANG=C.ASCII" or "LANG=C.latin-1" to get the
old platform dependent non-UTF-8 behaviour back).

If we turned the warning off by default, but retained an easy
"on-switch" for the warning to help track places where locale coercion
is triggering unexpectedly (e.g. "PYTHONCOERCECLOCALE=warn"), that
would probably give us the best of both worlds.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com