Re: [Python-ideas] (no subject)

[Stephen J. Turnbull]

Victor Stinner writes:

 > Using a custom exception handler, you can run expensive functions,
 > like the feature: "suggest len when length is used".

LGTM.

 > The problem is then when students have to use a Python without the
 > custom exception handler.

Put the exception handler in an importable module in the stdlib.

Usual caveats about high bar, etc, but this should solve the "have
Python, no handler" issue going forward.

Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Better error messages [was: (no subject)]

[Matthias Bussonnier]

There are a couple of project that tried to improved heuristic on some
error messages.

Two I can think off are:
   https://github.com/SylvainDe/DidYouMean-Python
and
   https://github.com/dutc/didyoumean

I think that better error messages could be implemented only in the
repl, and/or by alternative repl. Having it as an opt-in library would
be useful to tweak the messages in various context, for example when
used for teaching, or if there are concern about overhead.

Some alternative even already implement heuristic (Recursions error in
IPython are elided with "... last  frames repeated, from the frame
below ..." which is way less scarier for beginners. Some other
attempts have been made with attributes error[1] (similar to what
Stephen is proposing), though the traceback info does not have a ref
(or even weakref) on the object on which the attributes access is
made, so even more heuristic have to be made.

Maybe just adding some more information on exception themselves and
giving the recipe of custom except hook would be enough.
Then just allow users to pip-install packages with heuristics ?


-- 
M
[1] https://github.com/ipython/ipython/pull/9073

On Tue, Nov 29, 2016 at 6:14 PM, Stephen J. Turnbull
 wrote:
> Mariatta Wijaya writes:
>
>  > > NameError: name 'length' is not defined
>  >
>  > > A better message might be:
>  >
>  > > Python doesn't recognise the function "length". Did you mean
>  > > len?'
>
> This particular change would be useful to a beginning Python
> programmer.  I've made that error often enough myself (maybe I'm not a
> good example, though, in Lisp the generic function with the same role
> *is* called "length", so I make that error even today).
>
> But I wonder if it's a good example for the generic case.  An error
> message of the form 'Python doesn't recognize the function "".
> Did you mean ""?' could easily be misleading.  Python has
> functions, but they're a type of object.  "length" is a name, which is
> not an object.  The expected type of the object is not function, it's
> callable.  So consider:
>
> class Blog:
> pass
>
> blog = log()
>
> NameError: Python doesn't recognize the function "log".  Did you
> mean "Blog"?
>
> I suspect that might cause cognitive dissonance in a beginner who
> thinks of a class as a type or data structure and not as a function.
> And "doesn't recognize the callable" might be awkward (or beginners
> might get used to it very quickly, I don't know).
>
> Also, how do you propose deciding on the alternative to suggest?  In
> this particular case, I expect most developers would agree
> intuitively.  But the Hamming distance is pretty large: 3 of a length
> of 6.  Would you have a dictionary of common errors, and then scan the
> namespace for minimum Hamming distance among defined names with the
> right type of value?
>
> How about:
>
> class Blog:
> pass
>
> blog = get_blog_for_date(someday)
>
> logn = log(blog.size)
>
> NameError: Python doesn't recognize the function "log".  Did you
> mean "Blog"?
>
> Wouldn't
>
> NameError: Python doesn't recognize the name "log".  Perhaps
> you need to import the "math" module?
>
> be a better message here?  On second thought, that might imply that
> calling with the unqualified name is generally the best style, and
> teach the beginner to insert
>
> from math import *
>
> at the top of the module, thus fixing all such errors.  We probably
> don't want that, so maybe
>
> NameError: Python doesn't recognize the name "log".  There are
> functions named "log" in the "math" module and the "cmath" module.
>
> would be better yet.
>
> I definitely agree that there are times when Python's error messages
> are quite impenetrable for the beginner, and improvement is
> desirable.  I think that I would probably attack this by looking at
> the builtin namespace and a few stdlib namespaces (math, string, and
> maybe cmath come immediately to mind), and create a dictionary of
> "intuitive beginner errors".  Then trap NameErrors, and preferentially
> emit the message from the dictionary on exact matches.  Actually, come
> to think of it, two dictionaries, one for the builtin namespace, one
> for the selected stdlib, and the following heuristic:
>
> if name in common_builtin_typos:
> emit(common_builtin_typos[name])
> else:
> errors = small_hamming_distance(name, current_namespace, syntax(name))
> if errors:
> emit(errors)
> else:
> errors = exact_matches_in_imported_modules(name)
> if errors:
> emit(errors)
> elif name in common_unimported_stdlib_names:
> emit(common_unimported_stdlib_names[name])
> else:
> emit(error_you_would_have_emitted_anyway)
>
> In other words, I don't see a good systematic way to go about this,
> just pile up heuristics (and maybe remove some as they prove
> unuseful!)
>
> Steve
>
>
> 

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Nick Coghlan]

On 30 November 2016 at 04:33, Brett Cannon  wrote:
> On Tue, 29 Nov 2016 at 06:49 Nick Coghlan  wrote:
>>
>> On 29 November 2016 at 20:54, Tomas Orsava  wrote:
>> > With a metapath hook, .missing.py files are probably overkill, and the
>> > hook
>> > can just look at one file (or a static compiled-in list) of
>> > ModuleNotFound/ImportError messages for all missing modules, as M.-A.
>> > Lemburg and others are suggesting. We'll just need to think about
>> > coordinating how the list is generated/updated: the current PEP
>> > implicitly
>> > allows other parties, besides Python and the distributors, to step in
>> > cleanly if they need to—needing to update a single list could lead to
>> > messy
>> > hacks.
>>
>> What if, rather than using an explicitly file-based solution, this was
>> instead defined as a new protocol module, where the new metapath hook
>> imported a "__missing__" module and called a particular function in it
>> (e.g. "__missing__.module_not_found(modname)")?
>
>
> You can answer this question the best, Nick, but would it be worth defining
> a _stdlib.py that acts as both a marker for where the stdlib is installed --
> instead of os.py which is the current marker -- and which also stores
> metadata like an attribute called `missing` which is a dict that maps
> modules to ModuleNotFoundError messages? Although maybe this is too specific
> of a solution (or still too general and we use an e.g. missing.json off of
> sys.path which contains the same mapping).

Really, I think the ideal solution from a distro perspective would be
to enable something closer to what bash and other shells support for
failed CLI calls:

$ blender
bash: blender: command not found...
Install package 'blender' to provide command 'blender'? [N/y] n

This would allow redistributors to point folks towards platform
packages (via apt/yum/dnf/PyPM/conda/Canopy/etc) for the components
they provide, and towards pip/PyPI for everything else (and while we
don't have a dist-lookup-by-module-name service for PyPI *today*, it's
something I hope we'll find a way to provide sometime in the next few
years).

I didn't suggest that during the Fedora-level discussions of this PEP
because it didn't occur to me - the elegant simplicity of the new
import suffix as a tactical solution to the immediate "splitting the
standard library" problem [1] meant I missed that it was really a
special case of the general "provide guidance on obtaining missing
modules from the system package manager" concept.

The problem with that idea however is that while it provides the best
possible interactive user experience, it's potentially really slow,
and hence too expensive to do for every import error - we would
instead need to find a way to run with Wolfgang Maier's suggestion of
only doing this for *unhandled* import errors.

Fortunately, we do have the appropriate mechanisms in place to support
that approach:

1. For interactive use, we have sys.excepthook
2. For non-interactive use, we have the atexit module

As a simple example of the former:

>>> def module_missing(modname):
... return f"Module not found: {modname}"
>>> def my_except_hook(exc_type, exc_value, exc_tb):
... if isinstance(exc_value, ModuleNotFoundError):
... print(module_missing(exc_value.name))
...
>>> sys.excepthook = my_except_hook
>>> import foo
Module not found: foo
>>> import foo.bar
Module not found: foo
>>> import sys.bar
Module not found: sys.bar

For the atexit handler, that could be installed by the `site` module,
so the existing mechanisms for disabling site module processing would
also disable any default exception reporting hooks. Folks could also
register their own handlers via either `sitecustomize.py` or
`usercustomize.py`.

And at that point the problem starts looking less like "Customise the
handling of missing modules" and more like "Customise the rendering
and reporting of particular types of unhandled exceptions". For
example, a custom handler for subprocess.CalledProcessError could
introspect the original command and use `shutil.which` to see if the
requested command was even visible from the current process (and, in a
redistributor provided Python, indicate which system packages to
install to obtain the requested command).

> My personal vote is a callback called at
> https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L948
> with a default implementation that raises ModuleNotFoundError just like the
> current line does.

Ethan's observation about try/except import chains has got me think
that limiting this to handling errors within the context of single
import statement will be problematic, especially given that folks can
already write their own metapath hook for that case if they really
want to.

Cheers,
Nick.

[1] For folks wondering "This problem has existed for years, why
suddenly worry about it now?", Fedora's in the process of splitting
out an even more restr

[Python-ideas] Better error messages [was: (no subject)]

[Stephen J. Turnbull]

Mariatta Wijaya writes:

 > > NameError: name 'length' is not defined
 > 
 > > A better message might be:
 > 
 > > Python doesn't recognise the function "length". Did you mean
 > > len?'

This particular change would be useful to a beginning Python
programmer.  I've made that error often enough myself (maybe I'm not a
good example, though, in Lisp the generic function with the same role
*is* called "length", so I make that error even today).

But I wonder if it's a good example for the generic case.  An error
message of the form 'Python doesn't recognize the function "".
Did you mean ""?' could easily be misleading.  Python has
functions, but they're a type of object.  "length" is a name, which is
not an object.  The expected type of the object is not function, it's
callable.  So consider:

class Blog:
pass

blog = log()

NameError: Python doesn't recognize the function "log".  Did you
mean "Blog"?

I suspect that might cause cognitive dissonance in a beginner who
thinks of a class as a type or data structure and not as a function.
And "doesn't recognize the callable" might be awkward (or beginners
might get used to it very quickly, I don't know).

Also, how do you propose deciding on the alternative to suggest?  In
this particular case, I expect most developers would agree
intuitively.  But the Hamming distance is pretty large: 3 of a length
of 6.  Would you have a dictionary of common errors, and then scan the
namespace for minimum Hamming distance among defined names with the
right type of value?

How about:

class Blog:
pass

blog = get_blog_for_date(someday)

logn = log(blog.size)

NameError: Python doesn't recognize the function "log".  Did you
mean "Blog"?

Wouldn't

NameError: Python doesn't recognize the name "log".  Perhaps
you need to import the "math" module?

be a better message here?  On second thought, that might imply that
calling with the unqualified name is generally the best style, and
teach the beginner to insert

from math import *

at the top of the module, thus fixing all such errors.  We probably
don't want that, so maybe

NameError: Python doesn't recognize the name "log".  There are
functions named "log" in the "math" module and the "cmath" module.

would be better yet.

I definitely agree that there are times when Python's error messages
are quite impenetrable for the beginner, and improvement is
desirable.  I think that I would probably attack this by looking at
the builtin namespace and a few stdlib namespaces (math, string, and
maybe cmath come immediately to mind), and create a dictionary of
"intuitive beginner errors".  Then trap NameErrors, and preferentially
emit the message from the dictionary on exact matches.  Actually, come
to think of it, two dictionaries, one for the builtin namespace, one
for the selected stdlib, and the following heuristic:

if name in common_builtin_typos:
emit(common_builtin_typos[name])
else:
errors = small_hamming_distance(name, current_namespace, syntax(name))
if errors:
emit(errors)
else:
errors = exact_matches_in_imported_modules(name)
if errors:
emit(errors)
elif name in common_unimported_stdlib_names:
emit(common_unimported_stdlib_names[name])
else:
emit(error_you_would_have_emitted_anyway)

In other words, I don't see a good systematic way to go about this,
just pile up heuristics (and maybe remove some as they prove
unuseful!)

Steve


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Eric V. Smith]

On 11/29/2016 1:33 PM, Brett Cannon wrote:



On Tue, 29 Nov 2016 at 06:49 Nick Coghlan mailto:ncogh...@gmail.com>> wrote:

On 29 November 2016 at 20:54, Tomas Orsava mailto:tors...@redhat.com>> wrote:
> With a metapath hook, .missing.py files are probably overkill, and
the hook
> can just look at one file (or a static compiled-in list) of
> ModuleNotFound/ImportError messages for all missing modules, as M.-A.
> Lemburg and others are suggesting. We'll just need to think about
> coordinating how the list is generated/updated: the current PEP
implicitly
> allows other parties, besides Python and the distributors, to step in
> cleanly if they need to—needing to update a single list could lead
to messy
> hacks.

What if, rather than using an explicitly file-based solution, this was
instead defined as a new protocol module, where the new metapath hook
imported a "__missing__" module and called a particular function in it
(e.g. "__missing__.module_not_found(modname)")?


You can answer this question the best, Nick, but would it be worth
defining a _stdlib.py that acts as both a marker for where the stdlib is
installed -- instead of os.py which is the current marker -- and which
also stores metadata like an attribute called `missing` which is a dict
that maps modules to ModuleNotFoundError messages? Although maybe this
is too specific of a solution (or still too general and we use an e.g.
missing.json off of sys.path which contains the same mapping).

Otherwise MAL touched on the solution I always had in the back of my
head where we let people register a callback that gets passed the name
of any module that wasn't found through sys.meta_path. We could either
have the return value mean nothing and by default raise
ModuleNotFoundError, have the return value be what to set the module to
and raise an exception as expected, or have it be more error-specific
and return an exception to raise (all of these options also ask whether
a default callback doing what is normal is provided or if it's None by
default and import continues to provide the default semantics). The perk
of the callback is it removes the order sensitivity of any sys.meta_path
or sys.path_hooks solution where people might be doing
sys.meta_path.append(custom_finder) and thus won't necessarily trigger
if a new hook for missing modules is put at the end by default.


How about having a sys.meta_path_last_chance (or whatever), which is 
identical to anything else on sys.meta_path, but is always guaranteed to 
be called last.


Then instead of:
meta_path = sys.meta_path

it would be:
meta_path = sys.meta_path + ([sys.meta_path_last_chance] if 
sys.meta_path_last_chance else [])


There's no need to invent a new callback signature, or any new logic 
anywhere: it's literally just another metapath importer, but always 
guaranteed to be last.


Eric.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Yury Selivanov]

I actually thought about adding 'Exception.__hint__' attribute which 
would have a longer message explaining what happened and how to fix it.  
displayhooks can be easily modified to print __hint__ when it's set.  We 
can then add C API to set hints.



To address any possible performance concerns, we can disable hints in -O 
mode (essentially we can make them almost zero-cost).



Yury

On 2016-11-29 12:43 PM, Brett Cannon wrote:

On Tue, 29 Nov 2016 at 02:39 Nathaniel Smith  wrote:


On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner
 wrote:

Hi,

Python is optimized for performance. Formatting an error message has a
cost on performances.

Sure, but we have to look at this on a case-by-case basis. Is there
really important code out there that's generating NameErrors or
SyntaxErrors in an inner loop? That seems unlikely to me.

Even IndexError I'm a bit skeptical about. I can believe that there's
code that intentionally generates and then catches IndexError, but
AttributeError in my experience is much more performance-sensitive
than IndexError, because every failed hasattr call allocates an
AttributeError and hasattr is commonly used for feature checks. Yet
AttributeError has a much more informative (= expensive) message than
IndexError:

In [1]: object().a
AttributeError: 'object' object has no attribute 'a'

In [2]: list()[0]
IndexError: list index out of range


One way to make this cheap is to have a reasonable default message and use
attributes on the exceptions trigger the use of the default message. Nearly
a year ago I filed a bunch of issues for ideas on providing attributes on
exceptions where it made sense, e.g. an index attribute on IndexError (
http://bugs.python.org/issue18162). If we did this then for classes like
IndexError there constructor could be `IndexError(index=10, start=0,
end=3)` and then __str__() can lazily construct the string representation
using a default message, e.g. `"index {} is out of range of{} to
{}".format(index, start, end)`. Make the arguments keyword-only and they
become backwards-compatible and so the only overhead you pay for these
richer messages are keyword-based construction if you simply never access
the repr for the exception.



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[MRAB]

On 2016-11-29 19:45, Brendan Barnwell wrote:

On 2016-11-29 09:43, Brett Cannon wrote:

One way to make this cheap is to have a reasonable default message and
use attributes on the exceptions trigger the use of the default message.
Nearly a year ago I filed a bunch of issues for ideas on providing
attributes on exceptions where it made sense, e.g. an index attribute on
IndexError (http://bugs.python.org/issue18162). If we did this then for
classes like IndexError there constructor could be `IndexError(index=10,
start=0, end=3)` and then __str__() can lazily construct the string
representation using a default message, e.g. `"index {} is out of range
of{} to {}".format(index, start, end)`. Make the arguments keyword-only
and they become backwards-compatible and so the only overhead you pay
for these richer messages are keyword-based construction if you simply
never access the repr for the exception.


I absolutely think this is the way to go.  Having the relevant
information (the list that was too short, the index that was too big,
the key that wasn't there, etc.) is useful in many situations, and it's
much better to have that information in a programmatic form than just
squashed into an error message.  This then makes it relatively easy to
write wrappers that take bubbling-up exceptions and try to construct
more detailed messages for a less experienced audience.  Right now this
is difficult or impossible because the exception objects don't record
the information that would be needed for these expanded messages.

Couldn't that result in objects being held for longer, taking up memory, 
not being collected as promptly, and not releasing resources as quickly?


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Nathaniel Smith]

On Nov 29, 2016 9:43 AM, "Brett Cannon"  wrote:
>
>
>
> On Tue, 29 Nov 2016 at 02:39 Nathaniel Smith  wrote:
>>
>> On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner
>>  wrote:
>> > Hi,
>> >
>> > Python is optimized for performance. Formatting an error message has a
>> > cost on performances.
>>
>> Sure, but we have to look at this on a case-by-case basis. Is there
>> really important code out there that's generating NameErrors or
>> SyntaxErrors in an inner loop? That seems unlikely to me.
>>
>> Even IndexError I'm a bit skeptical about. I can believe that there's
>> code that intentionally generates and then catches IndexError, but
>> AttributeError in my experience is much more performance-sensitive
>> than IndexError, because every failed hasattr call allocates an
>> AttributeError and hasattr is commonly used for feature checks. Yet
>> AttributeError has a much more informative (= expensive) message than
>> IndexError:
>>
>> In [1]: object().a
>> AttributeError: 'object' object has no attribute 'a'
>>
>> In [2]: list()[0]
>> IndexError: list index out of range
>
>
> One way to make this cheap is to have a reasonable default message and
use attributes on the exceptions trigger the use of the default message.
Nearly a year ago I filed a bunch of issues for ideas on providing
attributes on exceptions where it made sense, e.g. an index attribute on
IndexError (http://bugs.python.org/issue18162). If we did this then for
classes like IndexError there constructor could be `IndexError(index=10,
start=0, end=3)` and then __str__() can lazily construct the string
representation using a default message, e.g. `"index {} is out of range
of{} to {}".format(index, start, end)`. Make the arguments keyword-only and
they become backwards-compatible and so the only overhead you pay for these
richer messages are keyword-based construction if you simply never access
the repr for the exception.

It seems like this might need some care, though, to make sure that these
extra attributes don't end up pinning objects in memory that shouldn't be?
Actually I always assumed that was why AttributeError's message was
constructed eagerly...

-n
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Terry Reedy]

On 11/29/2016 11:32 AM, Rob Cliffe wrote:



On 29/11/2016 04:58, victor rajewski wrote:


Traceback (most recent call last):

 File "foo.py", line 2, in 

   l[10]=14

IndexError: list assignment index out of range


A better message might be:

You tried to use l[10] when l is only 4 elements long. You can add
items to l using l.append(value), or check your index value to make
sure that's really the position you wanted to access.




It would make sense to me to upgrade this particular error message to
IndexError: list assignment index 10 out of range 0 to 3 if it can
be done without too much difficulty or overhead.  (An empty list, and
perhaps one with only 1 element, would be special cases.)  Come to think
of it, is the word "assignment" needed?


It would help if the line were "l1[10] = 2 * l2[13] + 3".




--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Brendan Barnwell]

On 2016-11-29 09:43, Brett Cannon wrote:

One way to make this cheap is to have a reasonable default message and
use attributes on the exceptions trigger the use of the default message.
Nearly a year ago I filed a bunch of issues for ideas on providing
attributes on exceptions where it made sense, e.g. an index attribute on
IndexError (http://bugs.python.org/issue18162). If we did this then for
classes like IndexError there constructor could be `IndexError(index=10,
start=0, end=3)` and then __str__() can lazily construct the string
representation using a default message, e.g. `"index {} is out of range
of{} to {}".format(index, start, end)`. Make the arguments keyword-only
and they become backwards-compatible and so the only overhead you pay
for these richer messages are keyword-based construction if you simply
never access the repr for the exception.


	I absolutely think this is the way to go.  Having the relevant 
information (the list that was too short, the index that was too big, 
the key that wasn't there, etc.) is useful in many situations, and it's 
much better to have that information in a programmatic form than just 
squashed into an error message.  This then makes it relatively easy to 
write wrappers that take bubbling-up exceptions and try to construct 
more detailed messages for a less experienced audience.  Right now this 
is difficult or impossible because the exception objects don't record 
the information that would be needed for these expanded messages.


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."

   --author unknown
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Brett Cannon]

On Tue, 29 Nov 2016 at 10:28 Nick Timkovich  wrote:

> I would consider the speed of the "ultimate error handler" (i.e. whatever
> prints the traceback and kills the program) in the interpreter to be moot,
> so long as it takes a small fraction of a second. Optimizing Python's speed
> it crashes super-fast due to an *unhandled* NameError in your program seems
> folly.
>

So the performance worry isn't the traceback printer once the call stack
fully unwinds but the construction of the exception instance itself. E.g.
having to construct an expensive string for every instance of
AttributeError is more the worry than printing a traceback.


>
> Regarding more informative messages for (e.g.) IndexError, would those
> just apply to built-in types as they're the most universal, or should some
> additional introspection be done for similar ducks?
>
> If it's useful/there's interest, I could try to do some analysis of Python
> questions on SO and see what the most common errors are. I'd guess things
> like "'NoneType' object has no attribute ..." would probably be up there,
> but that's a whole can of worms as to why someone's trying to call a method
> on, index, etc. something they accidentally whacked (a = a.sort()).
>

I suspect if we decide to try and go with more informative messages with a
solution people are happy with then having an idea of where people get
tripped up regularly will be good to help focus the work.

And as for the NoneType issue specifically, there's an issue for that:
http://bugs.python.org/issue28702 .


>
> On Tue, Nov 29, 2016 at 11:42 AM, Chris Barker 
> wrote:
>
> On Tue, Nov 29, 2016 at 5:48 AM, Nick Coghlan  wrote:
>
> > SyntaxErrors in an inner loop? That seems unlikely to me.
>
>
> Syntax Errors are a special case, as by definition the code isn't being
> run yet (yes, there could be an eval in there...)
>
> So we could at least make those more informative without worrying about
> performance.
>
> Also -- would it be possible to tack on the more informative message at a
> higher level? Once the Exception bubbles up to the REPL, is there enough
> information available to make a more informative message?
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Brett Cannon]

On Tue, 29 Nov 2016 at 06:49 Nick Coghlan  wrote:

> On 29 November 2016 at 20:54, Tomas Orsava  wrote:
> > With a metapath hook, .missing.py files are probably overkill, and the
> hook
> > can just look at one file (or a static compiled-in list) of
> > ModuleNotFound/ImportError messages for all missing modules, as M.-A.
> > Lemburg and others are suggesting. We'll just need to think about
> > coordinating how the list is generated/updated: the current PEP
> implicitly
> > allows other parties, besides Python and the distributors, to step in
> > cleanly if they need to—needing to update a single list could lead to
> messy
> > hacks.
>
> What if, rather than using an explicitly file-based solution, this was
> instead defined as a new protocol module, where the new metapath hook
> imported a "__missing__" module and called a particular function in it
> (e.g. "__missing__.module_not_found(modname)")?
>

You can answer this question the best, Nick, but would it be worth defining
a _stdlib.py that acts as both a marker for where the stdlib is installed
-- instead of os.py which is the current marker -- and which also stores
metadata like an attribute called `missing` which is a dict that maps
modules to ModuleNotFoundError messages? Although maybe this is too
specific of a solution (or still too general and we use an e.g.
missing.json off of sys.path which contains the same mapping).

Otherwise MAL touched on the solution I always had in the back of my head
where we let people register a callback that gets passed the name of any
module that wasn't found through sys.meta_path. We could either have the
return value mean nothing and by default raise ModuleNotFoundError, have
the return value be what to set the module to and raise an exception as
expected, or have it be more error-specific and return an exception to
raise (all of these options also ask whether a default callback doing what
is normal is provided or if it's None by default and import continues to
provide the default semantics). The perk of the callback is it removes the
order sensitivity of any sys.meta_path or sys.path_hooks solution where
people might be doing sys.meta_path.append(custom_finder) and thus won't
necessarily trigger if a new hook for missing modules is put at the end by
default.

My personal vote is a callback called at
https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L948
with
a default implementation that raises ModuleNotFoundError just like the
current line does.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Nick Timkovich]

I would consider the speed of the "ultimate error handler" (i.e. whatever
prints the traceback and kills the program) in the interpreter to be moot,
so long as it takes a small fraction of a second. Optimizing Python's speed
it crashes super-fast due to an *unhandled* NameError in your program seems
folly.

Regarding more informative messages for (e.g.) IndexError, would those just
apply to built-in types as they're the most universal, or should some
additional introspection be done for similar ducks?

If it's useful/there's interest, I could try to do some analysis of Python
questions on SO and see what the most common errors are. I'd guess things
like "'NoneType' object has no attribute ..." would probably be up there,
but that's a whole can of worms as to why someone's trying to call a method
on, index, etc. something they accidentally whacked (a = a.sort()).

On Tue, Nov 29, 2016 at 11:42 AM, Chris Barker 
wrote:

> On Tue, Nov 29, 2016 at 5:48 AM, Nick Coghlan  wrote:
>
>> > SyntaxErrors in an inner loop? That seems unlikely to me.
>>
>
> Syntax Errors are a special case, as by definition the code isn't being
> run yet (yes, there could be an eval in there...)
>
> So we could at least make those more informative without worrying about
> performance.
>
> Also -- would it be possible to tack on the more informative message at a
> higher level? Once the Exception bubbles up to the REPL, is there enough
> information available to make a more informative message?
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Chris Barker]

On Tue, Nov 29, 2016 at 5:48 AM, Nick Coghlan  wrote:

> > SyntaxErrors in an inner loop? That seems unlikely to me.
>

Syntax Errors are a special case, as by definition the code isn't being run
yet (yes, there could be an eval in there...)

So we could at least make those more informative without worrying about
performance.

Also -- would it be possible to tack on the more informative message at a
higher level? Once the Exception bubbles up to the REPL, is there enough
information available to make a more informative message?

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Brett Cannon]

On Tue, 29 Nov 2016 at 02:39 Nathaniel Smith  wrote:

> On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner
>  wrote:
> > Hi,
> >
> > Python is optimized for performance. Formatting an error message has a
> > cost on performances.
>
> Sure, but we have to look at this on a case-by-case basis. Is there
> really important code out there that's generating NameErrors or
> SyntaxErrors in an inner loop? That seems unlikely to me.
>
> Even IndexError I'm a bit skeptical about. I can believe that there's
> code that intentionally generates and then catches IndexError, but
> AttributeError in my experience is much more performance-sensitive
> than IndexError, because every failed hasattr call allocates an
> AttributeError and hasattr is commonly used for feature checks. Yet
> AttributeError has a much more informative (= expensive) message than
> IndexError:
>
> In [1]: object().a
> AttributeError: 'object' object has no attribute 'a'
>
> In [2]: list()[0]
> IndexError: list index out of range
>

One way to make this cheap is to have a reasonable default message and use
attributes on the exceptions trigger the use of the default message. Nearly
a year ago I filed a bunch of issues for ideas on providing attributes on
exceptions where it made sense, e.g. an index attribute on IndexError (
http://bugs.python.org/issue18162). If we did this then for classes like
IndexError there constructor could be `IndexError(index=10, start=0,
end=3)` and then __str__() can lazily construct the string representation
using a default message, e.g. `"index {} is out of range of{} to
{}".format(index, start, end)`. Make the arguments keyword-only and they
become backwards-compatible and so the only overhead you pay for these
richer messages are keyword-based construction if you simply never access
the repr for the exception.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Steven D'Aprano]

On Tue, Nov 29, 2016 at 10:55:14AM -0500, Todd wrote:
> On Tue, Nov 29, 2016 at 4:13 AM, M.-A. Lemburg  wrote:

> > Just as with .pth files, the possibility to hook arbitrary code
> > execution into the module search path will get abused for
> > all kinds of weird things, esp. if the whole sys.path is
> > scanned for the .missing.py module and not only the part
> > where the stdlib lives (as was suggested in the thread).
> >
> > So why not limit the PEP to just the intended use case ?

> I think a better question is why should we artificially limit the PEP?

Because YAGNI.

Overly complex, complicated systems which do more than is needed 
"because it might be useful one day" is an anti-pattern. The intended 
use-case is to allow Linux distributions to customize the error message 
on ImportError. From there, it is a small step to allow *other* people 
to do the same thing. But it is a BIG step to go from that to a solution 
that executes arbitrary code.

Before we take that big step, we ought to have a good reason.


> These is something that could be useful outside of the stdlib.

Sure. I don't think there is any proposal to prevent people outside of 
Linux package distributors from using this mechanism. I'm not sure how 
this would even be possible: if Red Hat or Debian can create a .missing 
file, so can anyone else.


> At least
> for Linux packages it is common to split out optional components of a
> python package into separate linux packages to limit the size and
> dependencies of the main package.  This could help a lot in that situation.

Well... I'm not sure how common that it. But it doesn't really matter.

This is a good argument for having a separate .missing file for each 
module, rather than a single flat registry of custom error messages. 
Separate .missing files will allow any Python package to easily install 
their own message. But either way, whether there's a single registry or 
an import hook that searches for .missing files if and only if the 
import failed, I haven't seen a strong argument for allowing arbitrary 
Python code.

(Earlier I suggested such a flat registry -- I now withdraw that 
suggestion. I'm satisfied that a separate spam.missing file containing 
the custom error message when spam.py cannot be found is a better way to 
handle this.)


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Rob Cliffe]

On 29/11/2016 04:58, victor rajewski wrote:


Traceback (most recent call last):

 File "foo.py", line 2, in 

   l[10]=14

IndexError: list assignment index out of range


A better message might be:

You tried to use l[10] when l is only 4 elements long. You can add 
items to l using l.append(value), or check your index value to make 
sure that's really the position you wanted to access.





It would make sense to me to upgrade this particular error message to
IndexError: list assignment index 10 out of range 0 to 3 if it can be 
done without too much difficulty or overhead.  (An empty list, and 
perhaps one with only 1 element, would be special cases.)  Come to think 
of it, is the word "assignment" needed?

Rob Cliffe
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Todd]

On Tue, Nov 29, 2016 at 4:13 AM, M.-A. Lemburg  wrote:

> On 29.11.2016 00:50, Brett Cannon wrote:
> > Seventh, these *.missing.py files if they are directly executed are
> totally
> > going to be abused like *.pth files, I can just feel it in my bones. We
> > need to be okay with this if we accept this PEP as-is.
>
> Since the purpose of the PEP was to allow distributors to guide
> users through the installation process of extra packages in order
> to get access to parts of the stdlib which are not installed,
> I think the PEP is overly broad in concept to address this one
> use case.
>
> Just as with .pth files, the possibility to hook arbitrary code
> execution into the module search path will get abused for
> all kinds of weird things, esp. if the whole sys.path is
> scanned for the .missing.py module and not only the part
> where the stdlib lives (as was suggested in the thread).
>
> So why not limit the PEP to just the intended use case ?
>
>
I think a better question is why should we artificially limit the PEP?
These is something that could be useful outside of the stdlib.  At least
for Linux packages it is common to split out optional components of a
python package into separate linux packages to limit the size and
dependencies of the main package.  This could help a lot in that situation.


> I.e. define a static list of modules which do make up the Python
> stdlib and then have the importer turn a ModuleNotFoundError error
> into a nice distribution specific error message, if and only
> if the imported module is from the set of stdlib modules.
>

This is hard to do in a general sense.  The point is to be able to tell the
user what package they should install to get that functionality, but there
is no general rule as to what the package should be named, and
platform-specific modules would not be installable at all.  So every module
would need its own error message defined.


>
> Thinking about this some more...
>
> We don't even need a list of stdlib modules. Simply define
> a general purpose import error formatting function, e.g.
> sys.formatimporterror(), pass in the name of the module and
> let it determine the error message based on the available
> information.
>
> A distributor could then provide a custom function that
> knows about the installed Python packages and then guides
> the user to install any missing ones.
>
>

 This is getting pretty complicated compared to simply defining a one-line
text file containing the error message with the module name somewhere in
the file name, as others have proposed.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Todd]

On Nov 29, 2016 5:51 AM, "Wolfgang Maier" <
wolfgang.ma...@biologie.uni-freiburg.de> wrote:
>
> On 29.11.2016 10:39, Paul Moore wrote:
>>
>> On 28 November 2016 at 22:33, Steve Dower  wrote:
>>>
>>> Given that, this wouldn't necessarily need to be an executable file. The
>>> finder could locate a "foo.missing" file and raise ModuleNotFoundError
with
>>> the contents of the file as the message. No need to allow/require any
Python
>>> code at all, and no risk of polluting sys.modules.
>>
>>
>> I like this idea. Would it completely satisfy the original use case
>> for the proposal? (Or, to put it another way, is there any specific
>> need for arbitrary code execution in the missing.py file?)
>>
>
> The only thing that I could think of so far would be cross-platform
.missing.py files that query the system (e.g. using the platform module) to
generate adequate messages for the specific platform or distro. E.g.,
correctly recommend to use dnf install or yum install or apt install, etc.

In those cases it would probably be as easy, if not easier, to do that at
build-time, which would get us back to simple text files.

Making a standard script is hard, if not impossible, in many cases because
the package name often does not match the module name.  So you are going to
need manual intervention in many cases, and modifying a one-line text file
is going to be easier than modifying a script.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Nick Coghlan]

On 29 November 2016 at 20:54, Tomas Orsava  wrote:
> With a metapath hook, .missing.py files are probably overkill, and the hook
> can just look at one file (or a static compiled-in list) of
> ModuleNotFound/ImportError messages for all missing modules, as M.-A.
> Lemburg and others are suggesting. We'll just need to think about
> coordinating how the list is generated/updated: the current PEP implicitly
> allows other parties, besides Python and the distributors, to step in
> cleanly if they need to—needing to update a single list could lead to messy
> hacks.

What if, rather than using an explicitly file-based solution, this was
instead defined as a new protocol module, where the new metapath hook
imported a "__missing__" module and called a particular function in it
(e.g. "__missing__.module_not_found(modname)")?

The default missing module implementation hook would just handle
CPython's optional modules, but redistributors could patch it to use a
mechanism that made sense for them.

For example, if we ever get to the point where the Fedora RPM database
includes "Provides: pythonXYimport(module.of.interest)" data in
addition to "Provides: pythonXYdist(pypi-package-name)" , the right
system package to import could be reported for any module, not just
standard library ones that have been split out (with the trade-off
being that any such checks would make optional imports a bit slower to
fail, but that could be mitigated in various ways).

Specific applications could also implement their own missing module
handling by providing a __missing__.py file alongside their
__main__.py, and relying on directory and/or zipfile execution, or
else by monkeypatching the __missing__ module at runtime.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Nick Coghlan]

On 29 November 2016 at 03:28, Guido van Rossum  wrote:
> On Mon, Nov 28, 2016 at 9:14 AM, Nathaniel Smith  wrote:
>>
>> Also note that in Guido's option 2, we only incur the extra fstat calls if
>> the import would otherwise fail. In option 1, there are extra fstat calls
>> (and thus disk seeks etc.) adding some overhead to every import.
>
> Oh, that's an important consideration! Yes, adding .missing.py to the list
> of extensions would cause extra stat() calls and potentially slow down every
> import.

This is the second time I've seen "but stat calls!" concern in
relation to import today, so I'll echo what Brett pointed out in his
reply: the import system in recent 3.x releases, along with the
importlib2 backport to Python 2.7, builds a cache of the directory
contents for path entries rather than making multiple stat calls.

The current 3.x source code for that is at
https://hg.python.org/cpython/file/tip/Lib/importlib/_bootstrap_external.py#l1280

The significant reduction in the number of stat calls through better
caching is the main way the 3.3 import reimplementation in Python
managed to be competitive performance-wise with the previous C
implementation, and faster when importing from a network filesystem :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Nick Coghlan]

On 29 November 2016 at 20:38, Nathaniel Smith  wrote:
> On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner
>  wrote:
>> Hi,
>>
>> Python is optimized for performance. Formatting an error message has a
>> cost on performances.
>
> Sure, but we have to look at this on a case-by-case basis. Is there
> really important code out there that's generating NameErrors or
> SyntaxErrors in an inner loop? That seems unlikely to me.

Right, we generally treat error message formatting code as being off
the critical performance path.

In many (most?) cases, the instances of uninformative error message
are just a symptom of the code in question being really *old*, such
that it predates the great many improvements made to the low level
error reporting machinery over the years.

That's not always true (e.g. parser errors are uninformative because
the parser doesn't keep track of the state needed to generate nicer
messages), but it seems to typically be true for runtime errors where
we don't even report the type or representation of a misbehaving
value.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Decorator to avoid a mistake

[Nick Coghlan]

On 29 November 2016 at 08:01, Chris Barker  wrote:
> On Mon, Nov 28, 2016 at 1:50 PM, Guido van Rossum  wrote:
>>>
>>> Also -- the ship has kinda sailed on this - maybe a @not_override would
>>> make more sense.
>>>
>>> Isn't the goal to make sure you don't accidentally override a method?
>>> saying "I know I'm overriding this" is less useful than "I'm not intending
>>> to override anything here"
>>
>> I think you're fighting a straw man. I never said @override should be
>> added to the language. I said that it would be useful to have a 3rd party
>> metaclass or a class decorator that implements it which packages may
>> voluntarily use to constrain their subclasses (or their own uses --
>> different designs are possible).
>
>
> I know -- I just happened to add that to a reply to you...
>
> That was for the OP, or anyone else thinking of writing such a thing.
>
> And I still think it would better be added to a linting tool than at
> run-time -- but let whoever writes it figure that out.

Writing linting tools for Python class hierarchies is pretty hard in
the general case, since you have to account for the fact that the
module level control flow logic is Turing complete (and many linters
simply don't try, instead saying "don't do that if you want the linter
to work properly").

By contrast, a class decorator or metaclass can do the checks at
runtime by walking the MRO, just as ABCMeta can look for
@abstractmethod declarations that haven't been overridden in a
subclass, and SQL Alchemy can map subclass inheritance to table
linkages.

If you look at the way attrs writes its automatic __init__ methods for
example, it creates a single flat __init__ based on the definition
time MRO: https://attrs.readthedocs.io/en/stable/how-does-it-work.html

That's the nice part about this kind of thing being opt-in: the
definition time tooling *doesn't* have to cope with the full extent of
Python's dynamic nature, as it can expect the user to be cooperating
with the tool to some degree, and "you're not cooperating" can be a
valid error to throw.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Paul Moore]

On 29 November 2016 at 10:51, Wolfgang Maier
 wrote:
> On 29.11.2016 10:39, Paul Moore wrote:
>>
>> On 28 November 2016 at 22:33, Steve Dower  wrote:
>>>
>>> Given that, this wouldn't necessarily need to be an executable file. The
>>> finder could locate a "foo.missing" file and raise ModuleNotFoundError
>>> with
>>> the contents of the file as the message. No need to allow/require any
>>> Python
>>> code at all, and no risk of polluting sys.modules.
>>
>>
>> I like this idea. Would it completely satisfy the original use case
>> for the proposal? (Or, to put it another way, is there any specific
>> need for arbitrary code execution in the missing.py file?)
>>
>
> The only thing that I could think of so far would be cross-platform
> .missing.py files that query the system (e.g. using the platform module) to
> generate adequate messages for the specific platform or distro. E.g.,
> correctly recommend to use dnf install or yum install or apt install, etc.

Yeah. I'd like to see a genuine example of how that would be used in
practice, otherwise I'd be inclined to suggest YAGNI. (Particularly
given that this PEP is simply a standardised means of vendor
customisation - for special cases, vendors obviously still have the
capability to patch or override standard behaviour in any way they
like).

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Wes Turner]

On Tuesday, November 29, 2016, Wes Turner  wrote:

> The existing docs for errors and exceptions:
>
> - https://docs.python.org/2/library/exceptions.html
> - https://docs.python.org/3/library/exceptions.html
> - https://hg.python.org/cpython/file/tip/Doc/library/exceptions.rst
> - https://github.com/python/cpython/blob/master/Doc/library/exceptions.rst
>
> - https://docs.python.org/2/tutorial/errors.html
> - https://docs.python.org/3/tutorial/errors.html
> - https://hg.python.org/cpython/file/tip/Doc/tutorial/errors.rst
> - https://github.com/python/cpython/blob/master/Doc/tutorial/errors.rst
>
> - https://www.tutorialspoint.com/python/python_exceptions.htm
>
> - If the docs don't answer the question (and match to the search terms),
> they probably should.
>
> - [ ] DOC: something about why "except Exception: pass" is usually bad
> - [ ] DOC: something about SystemExit and atexit: https://docs.python.
> org/2/library/atexit.html
>
>
> You can get alot more traceback from pytest (w/ pytest-sugar) and/or nose
> (with nose-progressive).
>
> There is extra information in the stack at exception time; but, IIUC, it
> would take a number of subclasses with class-specific docs and/or class
> introspection to be as detailed as "you probably wanted .append there
> because this is a List and the length is n but the key was".
>
> Maybe a "learning mode" which automatically calls inspect.getdoc() on
> Exception would be useful (sys.excepthook)?
> Practically, I usually just open an extra IPython shell and run
> `list.append?` for docs or `list.append??` for (Python but not C!) source
> (inspect.getsource).
> IPython also prints the function signature with `?`
>

- [ ] How could I also print out type annotations w/ function signatures
and docstrings?/??/???
https://github.com/python/typeshed/blob/master/stdlib/2/typing.pyi


>
> The pdb++ debugger requires funcsigs in order to print function
> signatures. If pdb++ is installed, it preempts the standard pdb module; so
> `nosetests --pdb` and `pytest --pdb` launch pdb++ when an error or
> exception is raised.
>
> https://pypi.python.org/pypi/pdbpp/
>
> http://nose.readthedocs.io/en/latest/plugins/debug.html
>
> http://doc.pytest.org/en/latest/usage.html
>
> https://docs.python.org/2/library/inspect.html
>
> Exceptions could be better someday. Testing (and debugging) skills are
> always good to learn; coincidentally, there are many great tools for it.
>
> ... https://westurner.org/wiki/awesome-python-testing#debugging
>
> On Tuesday, November 29, 2016, Victor Stinner  > wrote:
>
>> Hi,
>>
>> Python is optimized for performance. Formatting an error message has a
>> cost on performances.
>>
>> I suggest you to teach your student to use the REPL and use a custom
>> exception handler: sys.excepthook:
>> https://docs.python.org/2/library/sys.html#sys.excepthook
>>
>> Using a custom exception handler, you can run expensive functions,
>> like the feature: "suggest len when length is used".
>>
>> The problem is then when students have to use a Python without the
>> custom exception handler.
>>
>> Victor
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Wes Turner]

On Tuesday, November 29, 2016, Nathaniel Smith  wrote:

> On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner
> > wrote:
> > Hi,
> >
> > Python is optimized for performance. Formatting an error message has a
> > cost on performances.
>
> Sure, but we have to look at this on a case-by-case basis. Is there
> really important code out there that's generating NameErrors or
> SyntaxErrors in an inner loop? That seems unlikely to me.
>
> Even IndexError I'm a bit skeptical about. I can believe that there's
> code that intentionally generates and then catches IndexError, but
> AttributeError in my experience is much more performance-sensitive
> than IndexError, because every failed hasattr call allocates an
> AttributeError and hasattr is commonly used for feature checks. Yet
> AttributeError has a much more informative (= expensive) message than
> IndexError:
>
> In [1]: object().a
> AttributeError: 'object' object has no attribute 'a'
>
> In [2]: list()[0]
> IndexError: list index out of range


https://docs.python.org/2/tutorial/datastructures.html#more-on-lists

https://docs.python.org/2/c-api/list.html#c.PyList_SetItem

https://github.com/python/cpython/search?utf8=✓&q=PyList_SetItem

- https://github.com/python/cpython/blob/master/Include/listobject.h
- https://github.com/python/cpython/blob/master/Objects/listobject.c#L208
- https://hg.python.org/cpython/file/tip/Objects/listobject.c#l208

https://docs.python.org/3/library/collections.abc.html#collections-abstract-base-classes

- list (typeshed: List)
- Sequence
- MutableSequence
-

It would be great if these continue to match:
https://www.google.com/search?q=IndexError%3A+list+index+out+of+range



>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Python-ideas mailing list
> Python-ideas@python.org 
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Tomas Orsava]

On 11/28/2016 05:38 PM, Guido van Rossum wrote:

Overall I think this is a good idea. I have one hit:

It seems that there are two possible strategies for searching the 
.missing.py file:


1. (Currently in the PEP) search it at the same time as the .py file 
when walking along sys.path.
  - Pro: prevents confusion when the user accidentally has their own 
matching file later in sys.path.
  - Con: prevents the user from installing a matching file 
intentionally (e.g. a 3rd party version).


2. After exhausting sys.path, search it again just for .missing.py 
files (or perhaps remember the location of the .missing.py file during 
the first search but don't act immediately on it -- this has the same 
effect).

  - Pro: allows user to install their own version.
  - Con: if the user has a matching file by accident, that file will 
be imported, causing more confusion.


I personally would weigh these so as to prefer (2). The option of 
installing your own version when the standard version doesn't exist 
seems reasonable; there may be reasons that you can't or don't want to 
install the distribution's version. I don't worry much about the 
danger of accidental name conflicts (have you ever seen this?).


--Guido


Solution (2) is a very good alternative and can be implemented using a 
metapath hook as Steve proposed elsewhere in this thread [0].
We considered a similar metapath hook when designing the PEP, but 
decided against it, to better match the current behavior of third-party 
modules not being able to replace parts of stdlib.


Note that as Brett says elsewhere in the thread, due to caching there 
would be no extra stat() calls in the usual case. On the other hand, we 
aren't familiar with Windows, where replacing missing stdlib modules 
seems to be standard practice. Thanks for letting us know.


With a metapath hook, .missing.py files are probably overkill, and the 
hook can just look at one file (or a static compiled-in list) of 
ModuleNotFound/ImportError messages for all missing modules, as M.-A. 
Lemburg and others are suggesting. We'll just need to think about 
coordinating how the list is generated/updated: the current PEP 
implicitly allows other parties, besides Python and the distributors, to 
step in cleanly if they need to—needing to update a single list could 
lead to messy hacks.


We'll update the PEP to go with solution (2).


[0] https://mail.python.org/pipermail/python-ideas/2016-November/043837.html

Tomas Orsava




On Mon, Nov 28, 2016 at 8:13 AM, Paul Moore > wrote:


On 28 November 2016 at 15:51, Tomas Orsava mailto:tors...@redhat.com>> wrote:
> I believe I may have found the Windows curses implementation,
it's called
> PDCurses [0], and this website [1] appears to be distributing it
under the
> name `curses`.

My apologies, I should have included a pointer. That is indeed the
distribution I was thinking of.

> Could some Windows user please check if compiling Python with
the current
> reference implementation [2] of this PEP indeed generates a
> `curses.missing.py ` file among the
stdlib files? If so, we might consider
> skipping the generation of the .missing.py file for the curses
module on
> Windows.

I'll see if I can make some time to do the test. But as the change is
to setup.py, and the Windows build uses Visual Studio project files to
do the build, I expect that it won't generate missing.py files on
Windows. In actual fact, that may be the simplest solution, to note
that the build part of this change is restricted to Unix (non-Windows)
platforms specifically. As there's no real concept of a "distribution
version" of Python on Windows, it's probably not something that will
be that important on that platform (and support for .missing.py files
is there, it would just be necessary for distributors to manually
create those files as needed).

Paul
___
Python-ideas mailing list
Python-ideas@python.org 
https://mail.python.org/mailman/listinfo/python-ideas

Code of Conduct: http://python.org/psf/codeofconduct/





--
--Guido van Rossum (python.org/~guido )


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Wolfgang Maier]

On 29.11.2016 10:39, Paul Moore wrote:

On 28 November 2016 at 22:33, Steve Dower  wrote:

Given that, this wouldn't necessarily need to be an executable file. The
finder could locate a "foo.missing" file and raise ModuleNotFoundError with
the contents of the file as the message. No need to allow/require any Python
code at all, and no risk of polluting sys.modules.


I like this idea. Would it completely satisfy the original use case
for the proposal? (Or, to put it another way, is there any specific
need for arbitrary code execution in the missing.py file?)



The only thing that I could think of so far would be cross-platform 
.missing.py files that query the system (e.g. using the platform module) 
to generate adequate messages for the specific platform or distro. E.g., 
correctly recommend to use dnf install or yum install or apt install, etc.



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Cory Benfield]

> On 29 Nov 2016, at 02:48, Chris Angelico  wrote:
> 
> On Tue, Nov 29, 2016 at 12:14 PM, Steven D'Aprano  wrote:
>> What if I have two files?
>> 
>> # a.py
>> try:
>>import spam
>> except ImportError:
>>import ham as spam
>> 
>> # b.py
>> try:
>>import spam
>> except ImportError:
>>import cornedbeef as spam
>> 
> 
> In the same project? Then you already have a maintenance nightmare,
> because 'spam' will sometimes mean the same module (with state shared
> between the files), but might mean two distinct modules (and thus
> unrelated module objects). In different projects? They won't conflict.

Well, you *might* have a maintenance nightmare, but you might not.

In particular, I should point out that “spam” is just a name (more correctly 
referred to as a.spam and b.spam.) If the “spam” module is intended to have 
global state that the “a” and “b” modules use to communicate then obviously 
this is a problem. But if it isn’t, then there is exactly no problem with each 
module choosing its own fallback. As a really silly example, consider sqlite3 
again. If there were third-party modules that both implement the sqlite3 API, 
then there is no reason for each module to agree on what sqlite3 module they 
use unless types are being passed between them. If we consider “a” and “b” as 
truly separate non-communicating modules, then there’s no issue at all.

Cory
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Nathaniel Smith]

On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner
 wrote:
> Hi,
>
> Python is optimized for performance. Formatting an error message has a
> cost on performances.

Sure, but we have to look at this on a case-by-case basis. Is there
really important code out there that's generating NameErrors or
SyntaxErrors in an inner loop? That seems unlikely to me.

Even IndexError I'm a bit skeptical about. I can believe that there's
code that intentionally generates and then catches IndexError, but
AttributeError in my experience is much more performance-sensitive
than IndexError, because every failed hasattr call allocates an
AttributeError and hasattr is commonly used for feature checks. Yet
AttributeError has a much more informative (= expensive) message than
IndexError:

In [1]: object().a
AttributeError: 'object' object has no attribute 'a'

In [2]: list()[0]
IndexError: list index out of range

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Wes Turner]

The existing docs for errors and exceptions:

- https://docs.python.org/2/library/exceptions.html
- https://docs.python.org/3/library/exceptions.html
- https://hg.python.org/cpython/file/tip/Doc/library/exceptions.rst
- https://github.com/python/cpython/blob/master/Doc/library/exceptions.rst

- https://docs.python.org/2/tutorial/errors.html
- https://docs.python.org/3/tutorial/errors.html
- https://hg.python.org/cpython/file/tip/Doc/tutorial/errors.rst
- https://github.com/python/cpython/blob/master/Doc/tutorial/errors.rst

- https://www.tutorialspoint.com/python/python_exceptions.htm

- If the docs don't answer the question (and match to the search terms),
they probably should.

- [ ] DOC: something about why "except Exception: pass" is usually bad
- [ ] DOC: something about SystemExit and atexit:
https://docs.python.org/2/library/atexit.html


You can get alot more traceback from pytest (w/ pytest-sugar) and/or nose
(with nose-progressive).

There is extra information in the stack at exception time; but, IIUC, it
would take a number of subclasses with class-specific docs and/or class
introspection to be as detailed as "you probably wanted .append there
because this is a List and the length is n but the key was".

Maybe a "learning mode" which automatically calls inspect.getdoc() on
Exception would be useful (sys.excepthook)?
Practically, I usually just open an extra IPython shell and run
`list.append?` for docs or `list.append??` for (Python but not C!) source
(inspect.getsource).
IPython also prints the function signature with `?`

The pdb++ debugger requires funcsigs in order to print function signatures.
If pdb++ is installed, it preempts the standard pdb module; so `nosetests
--pdb` and `pytest --pdb` launch pdb++ when an error or exception is raised.

https://pypi.python.org/pypi/pdbpp/

http://nose.readthedocs.io/en/latest/plugins/debug.html

http://doc.pytest.org/en/latest/usage.html

https://docs.python.org/2/library/inspect.html

Exceptions could be better someday. Testing (and debugging) skills are
always good to learn; coincidentally, there are many great tools for it.

... https://westurner.org/wiki/awesome-python-testing#debugging

On Tuesday, November 29, 2016, Victor Stinner 
wrote:

> Hi,
>
> Python is optimized for performance. Formatting an error message has a
> cost on performances.
>
> I suggest you to teach your student to use the REPL and use a custom
> exception handler: sys.excepthook:
> https://docs.python.org/2/library/sys.html#sys.excepthook
>
> Using a custom exception handler, you can run expensive functions,
> like the feature: "suggest len when length is used".
>
> The problem is then when students have to use a Python without the
> custom exception handler.
>
> Victor
> ___
> Python-ideas mailing list
> Python-ideas@python.org 
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Paul Moore]

On 28 November 2016 at 22:33, Steve Dower  wrote:
> Given that, this wouldn't necessarily need to be an executable file. The
> finder could locate a "foo.missing" file and raise ModuleNotFoundError with
> the contents of the file as the message. No need to allow/require any Python
> code at all, and no risk of polluting sys.modules.

I like this idea. Would it completely satisfy the original use case
for the proposal? (Or, to put it another way, is there any specific
need for arbitrary code execution in the missing.py file?)

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[M.-A. Lemburg]

On 29.11.2016 00:50, Brett Cannon wrote:
> Seventh, these *.missing.py files if they are directly executed are totally
> going to be abused like *.pth files, I can just feel it in my bones. We
> need to be okay with this if we accept this PEP as-is.

Since the purpose of the PEP was to allow distributors to guide
users through the installation process of extra packages in order
to get access to parts of the stdlib which are not installed,
I think the PEP is overly broad in concept to address this one
use case.

Just as with .pth files, the possibility to hook arbitrary code
execution into the module search path will get abused for
all kinds of weird things, esp. if the whole sys.path is
scanned for the .missing.py module and not only the part
where the stdlib lives (as was suggested in the thread).

So why not limit the PEP to just the intended use case ?

I.e. define a static list of modules which do make up the Python
stdlib and then have the importer turn a ModuleNotFoundError error
into a nice distribution specific error message, if and only
if the imported module is from the set of stdlib modules.

The change of the error message could be done by having the
distributor patch the importer or we could have the importer
call a function defined via sitecustomize.py by the distributor
to return a message.

Thinking about this some more...

We don't even need a list of stdlib modules. Simply define
a general purpose import error formatting function, e.g.
sys.formatimporterror(), pass in the name of the module and
let it determine the error message based on the available
information.

A distributor could then provide a custom function that
knows about the installed Python packages and then guides
the user to install any missing ones.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 29 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (no subject)

[Victor Stinner]

Hi,

Python is optimized for performance. Formatting an error message has a
cost on performances.

I suggest you to teach your student to use the REPL and use a custom
exception handler: sys.excepthook:
https://docs.python.org/2/library/sys.html#sys.excepthook

Using a custom exception handler, you can run expensive functions,
like the feature: "suggest len when length is used".

The problem is then when students have to use a Python without the
custom exception handler.

Victor
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/