Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[M.-A. Lemburg]

On 07.12.2016 13:57, Nick Coghlan wrote:
> On 7 December 2016 at 18:33, M.-A. Lemburg  wrote:
>> I know that you started this thread focusing on the stdlib,
>> but for the purpose of distributors, the scope goes far
>> beyond just the stdlib.
>>
>> Basically any Python module or package which the distribution can
>> provide should be usable as basis for a nice error message pointing to
>> the package to install.
> 
> The PEP draft covered two questions:
> 
> - experienced redistributors breaking the standard library up into pieces
> - optional modules for folks building their own Python (even if
> they're new to that)
> 
>> Now, it's the distribution which knows which modules/packages
>> are available, so we don't need a list of stdlib modules
>> in Python to help with this.
> 
> Right, that's the case that we realised can be covered entirely by the
> suggestion "patch site.py to install a different default
> sys.excepthook()"
> 
>> A list of stdlib modules may still be useful, but it comes
>> with it's own set of problems, which should be irrelevant
>> for this use case: some stdlib modules are optional and
>> only available if the system provides (and Python can find)
>> certain libs (or header files during compilation).
> 
> While upstream changes turned out not to be necessary for the
> "distributor breaking up the standard library" use case, they may
> still prove worthwhile in making import errors more informative in the
> case of "I just built my own Python from upstream sources and didn't
> notice (or didn't read) the build message indicating that some modules
> weren't built".
> 
> Given the precedent of the sysconfig metadata generation, providing
> some form of machine-readable build-time-generated module manifest
> should be pretty feasible if someone was motivated to implement it,
> and we already have the logic to track which optional modules weren't
> built in order to generate the message at the end of the build
> process.

True, but the build process only covers C extensions. Writing
the information somewhere for Python to pick up would be easy,
though (just dump the .failed* lists somewhere).

For pure Python modules, I suppose the install process could
record all installed modules.

Put all this info into a generated "_sysconfigstdlib" module,
import this into sysconfig and you're set.

Still, in all the years I've been using Python I never ran
into a situation where I was interested in such information.

For cases where a module is optional, you usually write
a try...except and handle this on a case-by-case basis.

That's safer than relying on some build time generated
list, since the Python binary may well have been built
on a different machine than the one the application is
currently running on and so, even if an optional module
is listed as built successfully, it may still fail to
import.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Dec 07 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Stephen J. Turnbull]

Nick Coghlan writes:

 > While upstream changes turned out not to be necessary for the
 > "distributor breaking up the standard library" use case, they may
 > still prove worthwhile in making import errors more informative in the
 > case of "I just built my own Python from upstream sources and didn't
 > notice (or didn't read) the build message indicating that some modules
 > weren't built".

This case-by-case line of argument gives me a really bad feeling.  Do
we have to play whack-a-mole with every obscure message that pops up
that somebody might not be reading?  OK, this is a pretty common and
confusing case, but surely there's something more systematic (and
flexible vs. turning every error message into a complete usage manual
... which tl;dr) we can do.

One way to play would be an interactive checklist-based diagnostic
module (ie, a "rule-based expert system") that could be plugged into
IDEs or even into sys.excepthook.  Given Python's excellent
introspective facilities, with a little care the rule interpreter
could be designed with access to namespaces to provide additional
detail or tweak rule priority.  We could even build in a learning
engine to give priority to users' habitual bugs (including typical
mistaken diagnoses).

That said, I don't have time to work on it :-(, so feel free to ignore
me.  And I grant that since AFAIK we have zero existing code for the
engine and rule database, it might be a good idea to do something for
some particular obscure errors in the 3.7 timeframe.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Nick Coghlan]

On 7 December 2016 at 18:33, M.-A. Lemburg  wrote:
> I know that you started this thread focusing on the stdlib,
> but for the purpose of distributors, the scope goes far
> beyond just the stdlib.
>
> Basically any Python module or package which the distribution can
> provide should be usable as basis for a nice error message pointing to
> the package to install.

The PEP draft covered two questions:

- experienced redistributors breaking the standard library up into pieces
- optional modules for folks building their own Python (even if
they're new to that)

> Now, it's the distribution which knows which modules/packages
> are available, so we don't need a list of stdlib modules
> in Python to help with this.

Right, that's the case that we realised can be covered entirely by the
suggestion "patch site.py to install a different default
sys.excepthook()"

> A list of stdlib modules may still be useful, but it comes
> with it's own set of problems, which should be irrelevant
> for this use case: some stdlib modules are optional and
> only available if the system provides (and Python can find)
> certain libs (or header files during compilation).

While upstream changes turned out not to be necessary for the
"distributor breaking up the standard library" use case, they may
still prove worthwhile in making import errors more informative in the
case of "I just built my own Python from upstream sources and didn't
notice (or didn't read) the build message indicating that some modules
weren't built".

Given the precedent of the sysconfig metadata generation, providing
some form of machine-readable build-time-generated module manifest
should be pretty feasible if someone was motivated to implement it,
and we already have the logic to track which optional modules weren't
built in order to generate the message at the end of the build
process.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[M.-A. Lemburg]

I know that you started this thread focusing on the stdlib,
but for the purpose of distributors, the scope goes far
beyond just the stdlib.

Basically any Python module or package which the distribution can
provide should be usable as basis for a nice error message pointing to
the package to install.

Now, it's the distribution which knows which modules/packages
are available, so we don't need a list of stdlib modules
in Python to help with this.

The helper function (whether called via sys.excepthook() or perhaps
a new sys.importerrorhook()) would then check the imported
module name against this list and write out the message
pointing the user to the missing package.

A list of stdlib modules may still be useful, but it comes
with it's own set of problems, which should be irrelevant
for this use case: some stdlib modules are optional and
only available if the system provides (and Python can find)
certain libs (or header files during compilation).

For a distribution there are no optional stdlib modules,
since the distributor will know the complete list of available
modules in the distribution, including their external
dependencies.

In other words: Python already provides all the necessary
logic to enable implementing the suggested use case.


On 07.12.2016 06:24, Nick Coghlan wrote:
> On 7 December 2016 at 02:50, Tomas Orsava  wrote:
>> So using _sysconfigdata as inspiration, it would likely be possible to
>> provide a "sysconfig.get_missing_modules()" API that the default
>> sys.excepthook() could use to report that a particular import didn't
>> work because an optional standard library module hadn't been built.
>>
>> Quite interesting. And sysconfig.get_missing_modules() wouldn't even have to
>> be generated during the build process, because it would be called only when
>> the import has failed, at which point it is obvious Python was built without
>> said component (like _sqlite3). So do you see that as an acceptable
>> solution?
> 
> Oh, I'd missed that - yes, the sysconfig API could potentially be
> something like `sysconfig.get_stdlib_modules()` and
> `sysconfig.get_optional_modules()` instead of specifically reporting
> which ones were missed by the build process. There'd still be some
> work around generating the manifests backing those APIs at build time
> (including getting them right for Windows as well), but it would make
> some other questions that are currently annoying to answer relatively
> straightforward (see
> http://stackoverflow.com/questions/6463918/how-can-i-get-a-list-of-all-the-python-standard-library-modules
> for more on that)
> 
>> Do you prefer the one you suggested previously?
> 
> The only strong preference I have around how this is implemented is
> that I don't want to add complex single-purpose runtime infrastructure
> for the task. For all of the other specifics, I think it makes sense
> to err on the side of "What will be easiest to maintain over time?"
> 
>> Alternatively, can the contents of site.py be generated during the build
>> process? Because if some modules couldn't be built, a custom implementation
>> of sys.excepthook might be generated there with the data for the modules
>> that failed to be built.
> 
> We don't really want site.py itself to be auto-generated (although it
> could be updated to use Argument Clinic selectively if we deemed that
> to be an appropriate thing to do), but there's no problem with
> generating either data modules or normal importable modules that get
> accessed from site.py.
> 
> Cheers,
> Nick.
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Dec 07 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Tomas Orsava]

On 12/06/2016 03:27 AM, Nick Coghlan wrote:

On 5 December 2016 at 22:53, Tomas Orsava  wrote:

On 12/05/2016 01:42 PM, Nick Coghlan wrote:

Essentially, that would be the "name.missing.py" part of the draft
proposal for optional standard library modules, just with a regular
"name.py" module name and a tweak to getpath.c.
To my eye that looks like a complicated mechanism necessitating 
changes to

several parts of the codebase. Have you considered modifying the default
sys.excepthook implementation to read a list of modules and error 
messages
from a file that was generated during the build process? To me that 
seems

simpler, and the implementation will be only in one place.

In addition, distributors could just populate that file with their data,
thus we would have one mechanism for both use cases.

That's certainly another possibility, and one that initially appears
to confine most of the complexity to sys.excepthook(). However, the
problem you run into in that case is that CPython, by default, doesn't
have any configuration files other than site.py, sitecustomize.py,
usercustomize.py and whatever PYTHONSTARTUP points to for interactive
use. The only non-executable one that is currently defined is the
recommendation to redistributors in PEP 493 for file-based
configuration of HTTPS-verification-by-default backports to earlier
2.7.x versions.

Probably the closest analogy I can think of is the way we currently
generate _sysconfigdata-.py in order to
capture the build time settings such that sysconfig.get_config_vars()
can report them at runtime.

So using _sysconfigdata as inspiration, it would likely be possible to
provide a "sysconfig.get_missing_modules()" API that the default
sys.excepthook() could use to report that a particular import didn't
work because an optional standard library module hadn't been built.


Quite interesting. And sysconfig.get_missing_modules() wouldn't even 
have to be generated during the build process, because it would be 
called only when the import has failed, at which point it is obvious 
Python was built without said component (like _sqlite3). So do you see 
that as an acceptable solution? Do you prefer the one you suggested 
previously?


Alternatively, can the contents of site.py be generated during the build 
process? Because if some modules couldn't be built, a custom 
implementation of sys.excepthook might be generated there with the data 
for the modules that failed to be built.


Regards,
Tom
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Nick Coghlan]

On 5 December 2016 at 19:56, Tomas Orsava  wrote:
> On 12/03/2016 05:08 AM, Nick Coghlan wrote:
>>>
>>> Though I believe the default sys.excepthook function is currently written
>>> in
>>> C, so it wouldn't be very easy for distributors to customize it. Maybe it
>>> could be made to read module=error_message pairs from some external file,
>>> which would be easier to modify?
>>
>> The default implementation is written in C, but distributors could
>> patch site.py to replace it with a custom one written in Python. For
>> example, publish a "fedora-hooks" module to PyPI (so non-system Python
>> installations or applications regularly run without the site module
>> can readily use the same hooks if they choose to do so), and then
>> patch site.py in the system Python to do:
>>
>>  import fedora_hooks
>>  fedora_hooks.install_excepthook()
>>
>> The nice thing about that approach is it wouldn't need a new switch to
>> turn it off - it would get turned off with all the other site-specific
>> customisations when -S or -I is used. It would also better open things
>> up to redistributor experimentation in existing releases (2.7, 3.5,
>> etc) before we commit to a specific approach in the reference
>> interpreter (such as adding an optional 'platform.hooks' submodule
>> that vendors may provide, and relevant stdlib APIs will then call
>> automatically to override the default upstream provided processing).
>
> Ah, but of course! That leaves us with only one part of the PEP unresolved:
> When the build process is unable to compile some modules when building
> Python from source (such as _sqlite3 due to missing sqlite headers), it
> would be great to provide a custom message when one then tries to import
> such module when using the compiled Python.
>
> Do you see a 'pretty' solution for that within this framework?

I'm not sure it qualifies as 'pretty', but one approach would be to
have a './Modules/missing/' directory that gets pre-populated with
checked in ".py" files for extension modules that aren't always
built. When getpath.c detects it's running from a development
checkout, it would add that directory to sys.path (just before
site-packages), while 'make install' and 'make altinstall' would only
copy files from that directory into the installation target if the
corresponding extension modules were missing.

Essentially, that would be the "name.missing.py" part of the draft
proposal for optional standard library modules, just with a regular
"name.py" module name and a tweak to getpath.c.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Tomas Orsava]

On 12/05/2016 01:42 PM, Nick Coghlan wrote:

On 5 December 2016 at 19:56, Tomas Orsava  wrote:

On 12/03/2016 05:08 AM, Nick Coghlan wrote:

Though I believe the default sys.excepthook function is currently written
in
C, so it wouldn't be very easy for distributors to customize it. Maybe it
could be made to read module=error_message pairs from some external file,
which would be easier to modify?

The default implementation is written in C, but distributors could
patch site.py to replace it with a custom one written in Python. For
example, publish a "fedora-hooks" module to PyPI (so non-system Python
installations or applications regularly run without the site module
can readily use the same hooks if they choose to do so), and then
patch site.py in the system Python to do:

  import fedora_hooks
  fedora_hooks.install_excepthook()

The nice thing about that approach is it wouldn't need a new switch to
turn it off - it would get turned off with all the other site-specific
customisations when -S or -I is used. It would also better open things
up to redistributor experimentation in existing releases (2.7, 3.5,
etc) before we commit to a specific approach in the reference
interpreter (such as adding an optional 'platform.hooks' submodule
that vendors may provide, and relevant stdlib APIs will then call
automatically to override the default upstream provided processing).

Ah, but of course! That leaves us with only one part of the PEP unresolved:
When the build process is unable to compile some modules when building
Python from source (such as _sqlite3 due to missing sqlite headers), it
would be great to provide a custom message when one then tries to import
such module when using the compiled Python.

Do you see a 'pretty' solution for that within this framework?

I'm not sure it qualifies as 'pretty', but one approach would be to
have a './Modules/missing/' directory that gets pre-populated with
checked in ".py" files for extension modules that aren't always
built. When getpath.c detects it's running from a development
checkout, it would add that directory to sys.path (just before
site-packages), while 'make install' and 'make altinstall' would only
copy files from that directory into the installation target if the
corresponding extension modules were missing.

Essentially, that would be the "name.missing.py" part of the draft
proposal for optional standard library modules, just with a regular
"name.py" module name and a tweak to getpath.c.


To my eye that looks like a complicated mechanism necessitating changes 
to several parts of the codebase. Have you considered modifying the 
default sys.excepthook implementation to read a list of modules and 
error messages from a file that was generated during the build process? 
To me that seems simpler, and the implementation will be only in one place.


In addition, distributors could just populate that file with their data, 
thus we would have one mechanism for both use cases.


Tomas
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Tomas Orsava]

On 12/03/2016 05:08 AM, Nick Coghlan wrote:

Though I believe the default sys.excepthook function is currently written in
C, so it wouldn't be very easy for distributors to customize it. Maybe it
could be made to read module=error_message pairs from some external file,
which would be easier to modify?

The default implementation is written in C, but distributors could
patch site.py to replace it with a custom one written in Python. For
example, publish a "fedora-hooks" module to PyPI (so non-system Python
installations or applications regularly run without the site module
can readily use the same hooks if they choose to do so), and then
patch site.py in the system Python to do:

 import fedora_hooks
 fedora_hooks.install_excepthook()

The nice thing about that approach is it wouldn't need a new switch to
turn it off - it would get turned off with all the other site-specific
customisations when -S or -I is used. It would also better open things
up to redistributor experimentation in existing releases (2.7, 3.5,
etc) before we commit to a specific approach in the reference
interpreter (such as adding an optional 'platform.hooks' submodule
that vendors may provide, and relevant stdlib APIs will then call
automatically to override the default upstream provided processing).


Ah, but of course! That leaves us with only one part of the PEP 
unresolved: When the build process is unable to compile some modules 
when building Python from source (such as _sqlite3 due to missing sqlite 
headers), it would be great to provide a custom message when one then 
tries to import such module when using the compiled Python.


Do you see a 'pretty' solution for that within this framework?

Yours aye,
Tomas
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Nick Coghlan]

On 30 November 2016 at 04:33, Brett Cannon  wrote:
> On Tue, 29 Nov 2016 at 06:49 Nick Coghlan  wrote:
>>
>> On 29 November 2016 at 20:54, Tomas Orsava  wrote:
>> > With a metapath hook, .missing.py files are probably overkill, and the
>> > hook
>> > can just look at one file (or a static compiled-in list) of
>> > ModuleNotFound/ImportError messages for all missing modules, as M.-A.
>> > Lemburg and others are suggesting. We'll just need to think about
>> > coordinating how the list is generated/updated: the current PEP
>> > implicitly
>> > allows other parties, besides Python and the distributors, to step in
>> > cleanly if they need to—needing to update a single list could lead to
>> > messy
>> > hacks.
>>
>> What if, rather than using an explicitly file-based solution, this was
>> instead defined as a new protocol module, where the new metapath hook
>> imported a "__missing__" module and called a particular function in it
>> (e.g. "__missing__.module_not_found(modname)")?
>
>
> You can answer this question the best, Nick, but would it be worth defining
> a _stdlib.py that acts as both a marker for where the stdlib is installed --
> instead of os.py which is the current marker -- and which also stores
> metadata like an attribute called `missing` which is a dict that maps
> modules to ModuleNotFoundError messages? Although maybe this is too specific
> of a solution (or still too general and we use an e.g. missing.json off of
> sys.path which contains the same mapping).

Really, I think the ideal solution from a distro perspective would be
to enable something closer to what bash and other shells support for
failed CLI calls:

$ blender
bash: blender: command not found...
Install package 'blender' to provide command 'blender'? [N/y] n

This would allow redistributors to point folks towards platform
packages (via apt/yum/dnf/PyPM/conda/Canopy/etc) for the components
they provide, and towards pip/PyPI for everything else (and while we
don't have a dist-lookup-by-module-name service for PyPI *today*, it's
something I hope we'll find a way to provide sometime in the next few
years).

I didn't suggest that during the Fedora-level discussions of this PEP
because it didn't occur to me - the elegant simplicity of the new
import suffix as a tactical solution to the immediate "splitting the
standard library" problem [1] meant I missed that it was really a
special case of the general "provide guidance on obtaining missing
modules from the system package manager" concept.

The problem with that idea however is that while it provides the best
possible interactive user experience, it's potentially really slow,
and hence too expensive to do for every import error - we would
instead need to find a way to run with Wolfgang Maier's suggestion of
only doing this for *unhandled* import errors.

Fortunately, we do have the appropriate mechanisms in place to support
that approach:

1. For interactive use, we have sys.excepthook
2. For non-interactive use, we have the atexit module

As a simple example of the former:

>>> def module_missing(modname):
... return f"Module not found: {modname}"
>>> def my_except_hook(exc_type, exc_value, exc_tb):
... if isinstance(exc_value, ModuleNotFoundError):
... print(module_missing(exc_value.name))
...
>>> sys.excepthook = my_except_hook
>>> import foo
Module not found: foo
>>> import foo.bar
Module not found: foo
>>> import sys.bar
Module not found: sys.bar

For the atexit handler, that could be installed by the `site` module,
so the existing mechanisms for disabling site module processing would
also disable any default exception reporting hooks. Folks could also
register their own handlers via either `sitecustomize.py` or
`usercustomize.py`.

And at that point the problem starts looking less like "Customise the
handling of missing modules" and more like "Customise the rendering
and reporting of particular types of unhandled exceptions". For
example, a custom handler for subprocess.CalledProcessError could
introspect the original command and use `shutil.which` to see if the
requested command was even visible from the current process (and, in a
redistributor provided Python, indicate which system packages to
install to obtain the requested command).

> My personal vote is a callback called at
> https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L948
> with a default implementation that raises ModuleNotFoundError just like the
> current line does.

Ethan's observation about try/except import chains has got me think
that limiting this to handling errors within the context of single
import statement will be problematic, especially given that folks can
already write their own metapath hook for that case if they really
want to.

Cheers,
Nick.

[1] For folks wondering "This problem has existed for years, why
suddenly worry about it now?", 

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Eric V. Smith]

On 11/29/2016 1:33 PM, Brett Cannon wrote:



On Tue, 29 Nov 2016 at 06:49 Nick Coghlan > wrote:

On 29 November 2016 at 20:54, Tomas Orsava > wrote:
> With a metapath hook, .missing.py files are probably overkill, and
the hook
> can just look at one file (or a static compiled-in list) of
> ModuleNotFound/ImportError messages for all missing modules, as M.-A.
> Lemburg and others are suggesting. We'll just need to think about
> coordinating how the list is generated/updated: the current PEP
implicitly
> allows other parties, besides Python and the distributors, to step in
> cleanly if they need to—needing to update a single list could lead
to messy
> hacks.

What if, rather than using an explicitly file-based solution, this was
instead defined as a new protocol module, where the new metapath hook
imported a "__missing__" module and called a particular function in it
(e.g. "__missing__.module_not_found(modname)")?


You can answer this question the best, Nick, but would it be worth
defining a _stdlib.py that acts as both a marker for where the stdlib is
installed -- instead of os.py which is the current marker -- and which
also stores metadata like an attribute called `missing` which is a dict
that maps modules to ModuleNotFoundError messages? Although maybe this
is too specific of a solution (or still too general and we use an e.g.
missing.json off of sys.path which contains the same mapping).

Otherwise MAL touched on the solution I always had in the back of my
head where we let people register a callback that gets passed the name
of any module that wasn't found through sys.meta_path. We could either
have the return value mean nothing and by default raise
ModuleNotFoundError, have the return value be what to set the module to
and raise an exception as expected, or have it be more error-specific
and return an exception to raise (all of these options also ask whether
a default callback doing what is normal is provided or if it's None by
default and import continues to provide the default semantics). The perk
of the callback is it removes the order sensitivity of any sys.meta_path
or sys.path_hooks solution where people might be doing
sys.meta_path.append(custom_finder) and thus won't necessarily trigger
if a new hook for missing modules is put at the end by default.


How about having a sys.meta_path_last_chance (or whatever), which is 
identical to anything else on sys.meta_path, but is always guaranteed to 
be called last.


Then instead of:
meta_path = sys.meta_path

it would be:
meta_path = sys.meta_path + ([sys.meta_path_last_chance] if 
sys.meta_path_last_chance else [])


There's no need to invent a new callback signature, or any new logic 
anywhere: it's literally just another metapath importer, but always 
guaranteed to be last.


Eric.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Brett Cannon]

On Tue, 29 Nov 2016 at 06:49 Nick Coghlan  wrote:

> On 29 November 2016 at 20:54, Tomas Orsava  wrote:
> > With a metapath hook, .missing.py files are probably overkill, and the
> hook
> > can just look at one file (or a static compiled-in list) of
> > ModuleNotFound/ImportError messages for all missing modules, as M.-A.
> > Lemburg and others are suggesting. We'll just need to think about
> > coordinating how the list is generated/updated: the current PEP
> implicitly
> > allows other parties, besides Python and the distributors, to step in
> > cleanly if they need to—needing to update a single list could lead to
> messy
> > hacks.
>
> What if, rather than using an explicitly file-based solution, this was
> instead defined as a new protocol module, where the new metapath hook
> imported a "__missing__" module and called a particular function in it
> (e.g. "__missing__.module_not_found(modname)")?
>

You can answer this question the best, Nick, but would it be worth defining
a _stdlib.py that acts as both a marker for where the stdlib is installed
-- instead of os.py which is the current marker -- and which also stores
metadata like an attribute called `missing` which is a dict that maps
modules to ModuleNotFoundError messages? Although maybe this is too
specific of a solution (or still too general and we use an e.g.
missing.json off of sys.path which contains the same mapping).

Otherwise MAL touched on the solution I always had in the back of my head
where we let people register a callback that gets passed the name of any
module that wasn't found through sys.meta_path. We could either have the
return value mean nothing and by default raise ModuleNotFoundError, have
the return value be what to set the module to and raise an exception as
expected, or have it be more error-specific and return an exception to
raise (all of these options also ask whether a default callback doing what
is normal is provided or if it's None by default and import continues to
provide the default semantics). The perk of the callback is it removes the
order sensitivity of any sys.meta_path or sys.path_hooks solution where
people might be doing sys.meta_path.append(custom_finder) and thus won't
necessarily trigger if a new hook for missing modules is put at the end by
default.

My personal vote is a callback called at
https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L948
with
a default implementation that raises ModuleNotFoundError just like the
current line does.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Todd]

On Tue, Nov 29, 2016 at 4:13 AM, M.-A. Lemburg  wrote:

> On 29.11.2016 00:50, Brett Cannon wrote:
> > Seventh, these *.missing.py files if they are directly executed are
> totally
> > going to be abused like *.pth files, I can just feel it in my bones. We
> > need to be okay with this if we accept this PEP as-is.
>
> Since the purpose of the PEP was to allow distributors to guide
> users through the installation process of extra packages in order
> to get access to parts of the stdlib which are not installed,
> I think the PEP is overly broad in concept to address this one
> use case.
>
> Just as with .pth files, the possibility to hook arbitrary code
> execution into the module search path will get abused for
> all kinds of weird things, esp. if the whole sys.path is
> scanned for the .missing.py module and not only the part
> where the stdlib lives (as was suggested in the thread).
>
> So why not limit the PEP to just the intended use case ?
>
>
I think a better question is why should we artificially limit the PEP?
These is something that could be useful outside of the stdlib.  At least
for Linux packages it is common to split out optional components of a
python package into separate linux packages to limit the size and
dependencies of the main package.  This could help a lot in that situation.


> I.e. define a static list of modules which do make up the Python
> stdlib and then have the importer turn a ModuleNotFoundError error
> into a nice distribution specific error message, if and only
> if the imported module is from the set of stdlib modules.
>

This is hard to do in a general sense.  The point is to be able to tell the
user what package they should install to get that functionality, but there
is no general rule as to what the package should be named, and
platform-specific modules would not be installable at all.  So every module
would need its own error message defined.


>
> Thinking about this some more...
>
> We don't even need a list of stdlib modules. Simply define
> a general purpose import error formatting function, e.g.
> sys.formatimporterror(), pass in the name of the module and
> let it determine the error message based on the available
> information.
>
> A distributor could then provide a custom function that
> knows about the installed Python packages and then guides
> the user to install any missing ones.
>
>

 This is getting pretty complicated compared to simply defining a one-line
text file containing the error message with the module name somewhere in
the file name, as others have proposed.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Todd]

On Nov 29, 2016 5:51 AM, "Wolfgang Maier" <
wolfgang.ma...@biologie.uni-freiburg.de> wrote:
>
> On 29.11.2016 10:39, Paul Moore wrote:
>>
>> On 28 November 2016 at 22:33, Steve Dower  wrote:
>>>
>>> Given that, this wouldn't necessarily need to be an executable file. The
>>> finder could locate a "foo.missing" file and raise ModuleNotFoundError
with
>>> the contents of the file as the message. No need to allow/require any
Python
>>> code at all, and no risk of polluting sys.modules.
>>
>>
>> I like this idea. Would it completely satisfy the original use case
>> for the proposal? (Or, to put it another way, is there any specific
>> need for arbitrary code execution in the missing.py file?)
>>
>
> The only thing that I could think of so far would be cross-platform
.missing.py files that query the system (e.g. using the platform module) to
generate adequate messages for the specific platform or distro. E.g.,
correctly recommend to use dnf install or yum install or apt install, etc.

In those cases it would probably be as easy, if not easier, to do that at
build-time, which would get us back to simple text files.

Making a standard script is hard, if not impossible, in many cases because
the package name often does not match the module name.  So you are going to
need manual intervention in many cases, and modifying a one-line text file
is going to be easier than modifying a script.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Nick Coghlan]

On 29 November 2016 at 20:54, Tomas Orsava  wrote:
> With a metapath hook, .missing.py files are probably overkill, and the hook
> can just look at one file (or a static compiled-in list) of
> ModuleNotFound/ImportError messages for all missing modules, as M.-A.
> Lemburg and others are suggesting. We'll just need to think about
> coordinating how the list is generated/updated: the current PEP implicitly
> allows other parties, besides Python and the distributors, to step in
> cleanly if they need to—needing to update a single list could lead to messy
> hacks.

What if, rather than using an explicitly file-based solution, this was
instead defined as a new protocol module, where the new metapath hook
imported a "__missing__" module and called a particular function in it
(e.g. "__missing__.module_not_found(modname)")?

The default missing module implementation hook would just handle
CPython's optional modules, but redistributors could patch it to use a
mechanism that made sense for them.

For example, if we ever get to the point where the Fedora RPM database
includes "Provides: pythonXYimport(module.of.interest)" data in
addition to "Provides: pythonXYdist(pypi-package-name)" , the right
system package to import could be reported for any module, not just
standard library ones that have been split out (with the trade-off
being that any such checks would make optional imports a bit slower to
fail, but that could be mitigated in various ways).

Specific applications could also implement their own missing module
handling by providing a __missing__.py file alongside their
__main__.py, and relying on directory and/or zipfile execution, or
else by monkeypatching the __missing__ module at runtime.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Nick Coghlan]

On 29 November 2016 at 03:28, Guido van Rossum  wrote:
> On Mon, Nov 28, 2016 at 9:14 AM, Nathaniel Smith  wrote:
>>
>> Also note that in Guido's option 2, we only incur the extra fstat calls if
>> the import would otherwise fail. In option 1, there are extra fstat calls
>> (and thus disk seeks etc.) adding some overhead to every import.
>
> Oh, that's an important consideration! Yes, adding .missing.py to the list
> of extensions would cause extra stat() calls and potentially slow down every
> import.

This is the second time I've seen "but stat calls!" concern in
relation to import today, so I'll echo what Brett pointed out in his
reply: the import system in recent 3.x releases, along with the
importlib2 backport to Python 2.7, builds a cache of the directory
contents for path entries rather than making multiple stat calls.

The current 3.x source code for that is at
https://hg.python.org/cpython/file/tip/Lib/importlib/_bootstrap_external.py#l1280

The significant reduction in the number of stat calls through better
caching is the main way the 3.3 import reimplementation in Python
managed to be competitive performance-wise with the previous C
implementation, and faster when importing from a network filesystem :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Paul Moore]

On 29 November 2016 at 10:51, Wolfgang Maier
 wrote:
> On 29.11.2016 10:39, Paul Moore wrote:
>>
>> On 28 November 2016 at 22:33, Steve Dower  wrote:
>>>
>>> Given that, this wouldn't necessarily need to be an executable file. The
>>> finder could locate a "foo.missing" file and raise ModuleNotFoundError
>>> with
>>> the contents of the file as the message. No need to allow/require any
>>> Python
>>> code at all, and no risk of polluting sys.modules.
>>
>>
>> I like this idea. Would it completely satisfy the original use case
>> for the proposal? (Or, to put it another way, is there any specific
>> need for arbitrary code execution in the missing.py file?)
>>
>
> The only thing that I could think of so far would be cross-platform
> .missing.py files that query the system (e.g. using the platform module) to
> generate adequate messages for the specific platform or distro. E.g.,
> correctly recommend to use dnf install or yum install or apt install, etc.

Yeah. I'd like to see a genuine example of how that would be used in
practice, otherwise I'd be inclined to suggest YAGNI. (Particularly
given that this PEP is simply a standardised means of vendor
customisation - for special cases, vendors obviously still have the
capability to patch or override standard behaviour in any way they
like).

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Tomas Orsava]

On 11/28/2016 05:38 PM, Guido van Rossum wrote:

Overall I think this is a good idea. I have one hit:

It seems that there are two possible strategies for searching the 
.missing.py file:


1. (Currently in the PEP) search it at the same time as the .py file 
when walking along sys.path.
  - Pro: prevents confusion when the user accidentally has their own 
matching file later in sys.path.
  - Con: prevents the user from installing a matching file 
intentionally (e.g. a 3rd party version).


2. After exhausting sys.path, search it again just for .missing.py 
files (or perhaps remember the location of the .missing.py file during 
the first search but don't act immediately on it -- this has the same 
effect).

  - Pro: allows user to install their own version.
  - Con: if the user has a matching file by accident, that file will 
be imported, causing more confusion.


I personally would weigh these so as to prefer (2). The option of 
installing your own version when the standard version doesn't exist 
seems reasonable; there may be reasons that you can't or don't want to 
install the distribution's version. I don't worry much about the 
danger of accidental name conflicts (have you ever seen this?).


--Guido


Solution (2) is a very good alternative and can be implemented using a 
metapath hook as Steve proposed elsewhere in this thread [0].
We considered a similar metapath hook when designing the PEP, but 
decided against it, to better match the current behavior of third-party 
modules not being able to replace parts of stdlib.


Note that as Brett says elsewhere in the thread, due to caching there 
would be no extra stat() calls in the usual case. On the other hand, we 
aren't familiar with Windows, where replacing missing stdlib modules 
seems to be standard practice. Thanks for letting us know.


With a metapath hook, .missing.py files are probably overkill, and the 
hook can just look at one file (or a static compiled-in list) of 
ModuleNotFound/ImportError messages for all missing modules, as M.-A. 
Lemburg and others are suggesting. We'll just need to think about 
coordinating how the list is generated/updated: the current PEP 
implicitly allows other parties, besides Python and the distributors, to 
step in cleanly if they need to—needing to update a single list could 
lead to messy hacks.


We'll update the PEP to go with solution (2).


[0] https://mail.python.org/pipermail/python-ideas/2016-November/043837.html

Tomas Orsava




On Mon, Nov 28, 2016 at 8:13 AM, Paul Moore > wrote:


On 28 November 2016 at 15:51, Tomas Orsava > wrote:
> I believe I may have found the Windows curses implementation,
it's called
> PDCurses [0], and this website [1] appears to be distributing it
under the
> name `curses`.

My apologies, I should have included a pointer. That is indeed the
distribution I was thinking of.

> Could some Windows user please check if compiling Python with
the current
> reference implementation [2] of this PEP indeed generates a
> `curses.missing.py ` file among the
stdlib files? If so, we might consider
> skipping the generation of the .missing.py file for the curses
module on
> Windows.

I'll see if I can make some time to do the test. But as the change is
to setup.py, and the Windows build uses Visual Studio project files to
do the build, I expect that it won't generate missing.py files on
Windows. In actual fact, that may be the simplest solution, to note
that the build part of this change is restricted to Unix (non-Windows)
platforms specifically. As there's no real concept of a "distribution
version" of Python on Windows, it's probably not something that will
be that important on that platform (and support for .missing.py files
is there, it would just be necessary for distributors to manually
create those files as needed).

Paul
___
Python-ideas mailing list
Python-ideas@python.org 
https://mail.python.org/mailman/listinfo/python-ideas

Code of Conduct: http://python.org/psf/codeofconduct/





--
--Guido van Rossum (python.org/~guido )


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Wolfgang Maier]

On 29.11.2016 10:39, Paul Moore wrote:

On 28 November 2016 at 22:33, Steve Dower  wrote:

Given that, this wouldn't necessarily need to be an executable file. The
finder could locate a "foo.missing" file and raise ModuleNotFoundError with
the contents of the file as the message. No need to allow/require any Python
code at all, and no risk of polluting sys.modules.


I like this idea. Would it completely satisfy the original use case
for the proposal? (Or, to put it another way, is there any specific
need for arbitrary code execution in the missing.py file?)



The only thing that I could think of so far would be cross-platform 
.missing.py files that query the system (e.g. using the platform module) 
to generate adequate messages for the specific platform or distro. E.g., 
correctly recommend to use dnf install or yum install or apt install, etc.



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Cory Benfield]

> On 29 Nov 2016, at 02:48, Chris Angelico  wrote:
> 
> On Tue, Nov 29, 2016 at 12:14 PM, Steven D'Aprano  wrote:
>> What if I have two files?
>> 
>> # a.py
>> try:
>>import spam
>> except ImportError:
>>import ham as spam
>> 
>> # b.py
>> try:
>>import spam
>> except ImportError:
>>import cornedbeef as spam
>> 
> 
> In the same project? Then you already have a maintenance nightmare,
> because 'spam' will sometimes mean the same module (with state shared
> between the files), but might mean two distinct modules (and thus
> unrelated module objects). In different projects? They won't conflict.

Well, you *might* have a maintenance nightmare, but you might not.

In particular, I should point out that “spam” is just a name (more correctly 
referred to as a.spam and b.spam.) If the “spam” module is intended to have 
global state that the “a” and “b” modules use to communicate then obviously 
this is a problem. But if it isn’t, then there is exactly no problem with each 
module choosing its own fallback. As a really silly example, consider sqlite3 
again. If there were third-party modules that both implement the sqlite3 API, 
then there is no reason for each module to agree on what sqlite3 module they 
use unless types are being passed between them. If we consider “a” and “b” as 
truly separate non-communicating modules, then there’s no issue at all.

Cory
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Paul Moore]

On 28 November 2016 at 22:33, Steve Dower  wrote:
> Given that, this wouldn't necessarily need to be an executable file. The
> finder could locate a "foo.missing" file and raise ModuleNotFoundError with
> the contents of the file as the message. No need to allow/require any Python
> code at all, and no risk of polluting sys.modules.

I like this idea. Would it completely satisfy the original use case
for the proposal? (Or, to put it another way, is there any specific
need for arbitrary code execution in the missing.py file?)

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[M.-A. Lemburg]

On 29.11.2016 00:50, Brett Cannon wrote:
> Seventh, these *.missing.py files if they are directly executed are totally
> going to be abused like *.pth files, I can just feel it in my bones. We
> need to be okay with this if we accept this PEP as-is.

Since the purpose of the PEP was to allow distributors to guide
users through the installation process of extra packages in order
to get access to parts of the stdlib which are not installed,
I think the PEP is overly broad in concept to address this one
use case.

Just as with .pth files, the possibility to hook arbitrary code
execution into the module search path will get abused for
all kinds of weird things, esp. if the whole sys.path is
scanned for the .missing.py module and not only the part
where the stdlib lives (as was suggested in the thread).

So why not limit the PEP to just the intended use case ?

I.e. define a static list of modules which do make up the Python
stdlib and then have the importer turn a ModuleNotFoundError error
into a nice distribution specific error message, if and only
if the imported module is from the set of stdlib modules.

The change of the error message could be done by having the
distributor patch the importer or we could have the importer
call a function defined via sitecustomize.py by the distributor
to return a message.

Thinking about this some more...

We don't even need a list of stdlib modules. Simply define
a general purpose import error formatting function, e.g.
sys.formatimporterror(), pass in the name of the module and
let it determine the error message based on the available
information.

A distributor could then provide a custom function that
knows about the installed Python packages and then guides
the user to install any missing ones.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 29 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Chris Angelico]

On Tue, Nov 29, 2016 at 12:14 PM, Steven D'Aprano  wrote:
> What if I have two files?
>
> # a.py
> try:
> import spam
> except ImportError:
> import ham as spam
>
> # b.py
> try:
> import spam
> except ImportError:
> import cornedbeef as spam
>

In the same project? Then you already have a maintenance nightmare,
because 'spam' will sometimes mean the same module (with state shared
between the files), but might mean two distinct modules (and thus
unrelated module objects). In different projects? They won't conflict.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Ethan Furman]

On 11/28/2016 05:46 PM, Random832 wrote:

On Mon, Nov 28, 2016, at 15:05, Ethan Furman wrote:



Because it is unfriendly.  Helpful error messages are a great tool to
both beginner and seasoned programmers.


There won't be a helpful error message unless the distributor writes
one.


The purpose of this PEP, if I understand correctly, is to settle on a standard 
for the location of that distributor written helpful error message.  As a 
bonus, cpython itself can use the same mechanism for modules that are possible 
to build in the stdlib, but weren't.  This would be useful for folks that build 
their own version.


Whatever the standard method is, it has to be something we can direct
distributors to modify, it's simply not something Python can do on its
own.



---
Yo, Distributor!  If you move tkinter to a separate module, please add a 
tkinter.missing file in your main python package where tkinter is supposed to 
be; this file should contain a helpful message on how to install the tkinter 
package!
---

There.  Done.



 The default exception hook is as good a place as any. Maybe write
most of the logic and get the distributors to just populate an
empty-by-default array of structs with the module name and error message
(and what about localization?)


This might handle the stdlib portion, but the PEP's solution could be easily 
extended to handle any Python application that is installable in pieces.  As 
far as localization -- it's a small text file, surely there are mechanisms 
already to deal with that?  (I don't know, it's not a problem I have to deal 
with.)


And the idea that building a ".missing.py" for every optional module
that's disabled is going to adequate is a bit naive.


Who said disabled?  The PEP says missing, as in not there -- not disabled, as 
in there but ... what? not able to be used?


For one thing, they're not going to *be* disabled,


Ah, whew -- we agree on something! ;)


the distributors are going to build
the whole thing and break up the installed files into packages. And
you've still got to get the distributors to actually put their friendly
error message in those files,


No, we don't.  We provide a mechanism for them to use, and they use it or don't 
at their whim.  It's a quality-of-implementation issue.


and the missing.py files are build
artifacts instead of a source file that they can patch.


They are the ones that decide how to segment the stdlib, so they get to do the 
work.  I would imagine, or at least hope, that they have the build and 
segmentation code under version control -- they can patch that.

--
~Ethan~
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Steven D'Aprano]

On Tue, Nov 29, 2016 at 09:52:21AM +1100, Chris Angelico wrote:

> +1, because this also provides a coherent way to reword the try/except
> import idiom:
> 
> # Current idiom
> # somefile.py
> try:
> import foo
> except ImportError:
> import subst_foo as foo

Nice, clean and self-explanatory: import a module, if it fails, import 
its replacement. Obviously that idiom must die.

*wink*



> # New idiom:
> # foo.missing.py
> import subst_foo as foo
> import sys; sys.modules["foo"] = foo
> #somefile.py
> import foo

Black magic where the replacement happens out of sight.

What if I have two files?

# a.py
try:
import spam
except ImportError:
import ham as spam

# b.py
try:
import spam
except ImportError:
import cornedbeef as spam


The current idiom makes it the responsibility of the importer to decide 
what happens when the import fails. Perhaps it is as simple as importing 
a substitute module, but it might log a warning, monkey-patch some 
classes, who knows? The important thing is that the importer decides 
whether to fail or try something else.

Your proposal means that the importer no longer has any say in the 
matter. The administrator of the site chooses what .missing files get 
added, they decide whether or not to log a warning, they decide whether 
to substitute ham.py or cornedbeef.py for spam.

I do not like this suggestion, and the mere possibility that it could 
happen makes this entire .missing suggestion a very strong -1 from me.



To solve the simple problem of providing a more useful error message 
when an ImportError occurs, we do not need to execute arbitrary code in 
a .missing.py file. That's using a nuclear-powered bulldozer to crack a 
peanut.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Steven D'Aprano]

On Mon, Nov 28, 2016 at 12:05:01PM -0800, Ethan Furman wrote:

> >Honestly, though, I'm not sure of the need for the PEP in general.
> >"However, there is as of yet no standardized way of dealing with
> >importing a missing standard library module." is simply not true. The
> >standardized way of dealing with it is that the import statement will
> >raise an ImportError exception. Why exactly is that not good enough?
> 
> Because it is unfriendly.  Helpful error messages are a great tool to both 
> beginner and seasoned programmers.

Random already covers that. There's no reason why packagers can't fix 
that.


> >A distribution could, for example, include an excepthook in site.py that
> >prints an informative error message when an ImportError is unhandled for
> >a list of modules that it knows about.  [...]
> 
> As you say above, that list will fall out of date.  Better to have a 
> standard method that is easily implemented.

I think you have misunderstood Random's observation.

Random notes correctly that treating "curses on Windows" as a special 
case will get out of date. Today its curses, tomorrow it might be curses 
and foo, then curses foo and bar, then just foo. Who knows? And what 
about Linux and Mac users? Might we start deploying third-party 
replacements for Windows-only std lib modules? (If any.)

This is effectively a black-list:

- don't add a .missing file for these modules

where the list depends on guessing what *other* people do.

But Random's observation doesn't apply to the packager. They cannot fall 
out of date, since they're generating a *white-list* of modules they 
have split out of the std lib into a separate package. Instead of 
the packager doing this:

- remove foo, bar, baz from the standard python package;
- add foo.missing, bar.missing, baz.missing to the python package;
- add foo, bar, baz to the python-extras package

Random suggest that they do this:

- remove foo, bar, baz from the standard python package;
- add foo, bar, baz to the list of modules that ImportError knows about;
- add foo, bar, baz to the python-extras package.


It can no more get out of date than can the .missing files.

Instead of adding a complex mechanism for searching the PYTHONPATH 
twice, the second time looking for .missing files, here's a counter 
proposal: 

- Add a single config file in a known, standard place, call 
  it "missing.ini" for the sake of the argument;

- If present, that file should be a list of module names as keys
  and custom error messages as values;

  foo: try running "yum install spam-python"
  bar: try running "yum install spam-python"
  baz: try running "yum install eggs-python"


- When ImportError is raised, Python looks at that file, and if
  the module name is found, it gives the custom error message in
  addition to the regular error message:

  import foo

  ImportError: No module named 'foo'
  try running "yum install spam-python"


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Steven D'Aprano]

On Mon, Nov 28, 2016 at 02:32:16PM +, Paul Moore wrote:

> Also, and possibly more of an issue, use of the ".missing.py" file
> will mean that a user can't provide their own implementation of the
> module later on sys.path. I don'rt know if this is a significant issue
> on Unix platforms. On Windows, there is a 3rd party implementation of
> the curses module which (as I understand it) can be user installed. If
> Python included a curses.missing.py, that would no longer work.
> 
> Certainly these are only minor points, but worth considering.

I don't think that's a minor point, I think its a very important one.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Brett Cannon]

On Mon, 28 Nov 2016 at 14:49 Steve Dower  wrote:

> On 28Nov2016 1433, Steve Dower wrote:
> > On 28Nov2016 1419, Nathaniel Smith wrote:
> >> I'd suggest that we additional specify that if we find a
> >> foo.missing.py, then the code is executed but -- unlike a regular
> >> module load -- it's not automatically inserted into
> >> sys.modules["foo"]. That seems like it could only create confusion.
> >> And it doesn't restrict functionality, because if someone really wants
> >> to implement some clever shenanigans, they can always modify
> >> sys.modules["foo"] by hand.
> >
> > In before Brett says "you can do this with an import hook", because,
> > well, we can do this with an import hook :)
>
> And since I suggested it, here's a rough proof-of-concept:
>
> import importlib.abc
> import os
> import sys
>
> class MissingPathFinder(importlib.abc.MetaPathFinder):
>  def find_spec(self, fullname, path, target=None):
>  for p in (path or sys.path):
>  file = os.path.join(p, fullname + ".missing")
>  if os.path.isfile(file):
>  with open(file, 'r', encoding='utf-8') as f:
>  raise ModuleNotFoundError(f.read())
>
> sys.meta_path.append(MissingPathFinder())
> import foo
>
>
> Add a "foo.missing" file to your working directory and you'll get the
> message from that instead of the usual one.
>

Since this PEP directly affects import I'm going to weigh in.

First, this won't necessarily create more stat calls depending on how it's
implemented. importlib.machinery.FileFinder which does the searching for
files on sys.path caches directory contents for as long as the granularity
of the file system's mtime is (e.g. a 1 second mtime granularity means
directory contents are cached for 1 second). This means that if the check
occurs within that granularity (whether immediately after looking for *.py
or in some way through a second pass) then there's no file system overhead.

Second, as proposed the PEP probably shouldn't change
importlib.machinery.SourceFileLoader and instead should return some new
loader that only handles these *.missing.py files (just like a different
loader is returned for extension modules). This allows for the loader to be
simpler and avoids making any custom loader from no longer implementing
current semantics (although I have tried to structure things in importlib
to make it so subclassing is an attractive option for people so this isn't
vital, just something to at least consider).

Third, Steve channeled me properly and this actually doesn't require any
changes to any pre-existing code and can instead be implemented as an
importlib.abc.MetaPathFinder that is at the end of sys.meta_path which
means there wouldn't be any local shadowing of modules available farther
down sys.path (although this would lead to more stat calls).

Fourth, if you make a meta-path finder and use static data you do away with
any performance issue with the file system. And since it would be installed
at the end of the sys.meta_path -- and thus after
importlib.machinery.PathFinder -- you effectively shadow it with a
successful import and so there's no need to worry about the information
leaking out unless someone mucks with sys.meta_path.

Fifth, people have asked for some way to catch/log/manipulate the response
of import when a module isn't found, e.g. renaming modules in 2/3
migrations was the first major instance of this, but I've heard others
wanting to log this to detect what modules they should install for users in
a cloud environment. This is a specific solution to a general problem that
some have asked for so it might warrant thinking about whether a more
general solution could work (but never enough to warrant me trying to solve
it above other issues).

Sixth, this would be easier to deal with if import got refactored into its
own object and out of the sys.module for easier manipulation of the import
process. ;)

Seventh, these *.missing.py files if they are directly executed are totally
going to be abused like *.pth files, I can just feel it in my bones. We
need to be okay with this if we accept this PEP as-is.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Wolfgang Maier]

On 28.11.2016 23:52, Chris Angelico wrote:


+1, because this also provides a coherent way to reword the try/except
import idiom:

# Current idiom
# somefile.py
try:
import foo
except ImportError:
import subst_foo as foo

# New idiom:
# foo.missing.py
import subst_foo as foo
import sys; sys.modules["foo"] = foo
#somefile.py
import foo



Hmm. I would rather take this example as an argument against the 
proposed behavior. It invites too many clever hacks. I thought that the 
idea was that .missing.py does *not* act as a replacement module, but, 
more or less, just as a message generator.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Chris Angelico]

On Tue, Nov 29, 2016 at 9:19 AM, Nathaniel Smith  wrote:
> This also suggests that the overall error-handling flow for 'import
> foo' should look like:
>
> 1) run foo.missing.py
> 2) if it raises an exception: propagate that
> 3) otherwise, if sys.modules["foo"] is missing: raise some variety of
> ImportError.
> 4) otherwise, use sys.modules["foo"] as the object that should be
> bound to 'foo' in the original invoker's namespace

+1, because this also provides a coherent way to reword the try/except
import idiom:

# Current idiom
# somefile.py
try:
import foo
except ImportError:
import subst_foo as foo

# New idiom:
# foo.missing.py
import subst_foo as foo
import sys; sys.modules["foo"] = foo
#somefile.py
import foo

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Wolfgang Maier]

On 28.11.2016 23:19, Nathaniel Smith wrote:


I'd suggest that we additional specify that if we find a
foo.missing.py, then the code is executed but -- unlike a regular
module load -- it's not automatically inserted into
sys.modules["foo"]. That seems like it could only create confusion.
And it doesn't restrict functionality, because if someone really wants
to implement some clever shenanigans, they can always modify
sys.modules["foo"] by hand.

This also suggests that the overall error-handling flow for 'import
foo' should look like:

1) run foo.missing.py
2) if it raises an exception: propagate that
3) otherwise, if sys.modules["foo"] is missing: raise some variety of
ImportError.
4) otherwise, use sys.modules["foo"] as the object that should be
bound to 'foo' in the original invoker's namespace

I think this might make everyone who was worried about exception
handling downthread happy -- it allows a .missing.py file to
successfully import if it really wants to, but only if it explicitly
fulfills 'import' requirement that the module should somehow be made
available.



A refined (from my previous post which may have ended up too nested) 
alternative: instead of triggering an immediate search for a .missing.py 
file, why not have the interpreter intercept any ModuleNotFoundError 
that bubbles up to the top without being caught, then uses the name 
attribute of the exception to look for the .missing.py file. Agreed, 
this is more complicated to implement, but it would avoid any 
performance loss in situations where running code knows how to deal with 
the missing module anyway.


Wolfgang

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Nathaniel Smith]

On Mon, Nov 28, 2016 at 5:28 AM, Tomas Orsava  wrote:
[...]
> Specification
> =
>
> When, for any reason, a standard library module is not to be included with
> the
> rest, a file with its name and the extension ``.missing.py`` shall be
> created
> and placed in the directory the module itself would have occupied.  This
> file
> can contain any Python code, however, it *should* raise a
> ModuleNotFoundError_
> with a helpful error message.
>
> Currently, when Python tries to import a module ``XYZ``, the ``FileFinder``
> path hook goes through the entries in ``sys.path``, and in each location
> looks
> for a file whose name is ``XYZ`` with one of the valid suffixes (e.g.
> ``.so``,
> ..., ``.py``, ..., ``.pyc``).  The suffixes are tried in order.  If none of
> them are found, Python goes on to try the next directory in ``sys.path``.
>
> The ``.missing.py`` extension will be added to the end of the list, and
> configured to be handled by ``SourceFileLoader``.  Thus, if a module is not
> found in its proper location, the ``XYZ.missing.py`` file is found and
> executed, and further locations are not searched.

I'd suggest that we additional specify that if we find a
foo.missing.py, then the code is executed but -- unlike a regular
module load -- it's not automatically inserted into
sys.modules["foo"]. That seems like it could only create confusion.
And it doesn't restrict functionality, because if someone really wants
to implement some clever shenanigans, they can always modify
sys.modules["foo"] by hand.

This also suggests that the overall error-handling flow for 'import
foo' should look like:

1) run foo.missing.py
2) if it raises an exception: propagate that
3) otherwise, if sys.modules["foo"] is missing: raise some variety of
ImportError.
4) otherwise, use sys.modules["foo"] as the object that should be
bound to 'foo' in the original invoker's namespace

I think this might make everyone who was worried about exception
handling downthread happy -- it allows a .missing.py file to
successfully import if it really wants to, but only if it explicitly
fulfills 'import' requirement that the module should somehow be made
available.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Ethan Furman]

On 11/28/2016 01:26 PM, Paul Moore wrote:

On 28 November 2016 at 21:11, Ethan Furman  wrote:

One "successful" use-case that would be impacted is the fallback import
idiom:

try:
 # this would do two full searches before getting the error
 import BlahBlah
except ImportError:
 import blahblah


Under this proposal, the above idiom could potentially now fail. If
there's a BlahBlah.missing.py, then that will get executed rather than
an ImportError being raised, so the fallback wouldn't be executed.


Which is why the strong recommendation is for the .missing.py file to raise an 
ImportError exception, but with a useful error message, such as "Tkinter is not 
currently installed.  Install python-tkinter to get it."


This could actually be a serious issue for code that currently
protects against optional stdlib modules not being available like
this. There's no guarantee that I can see that a .missing.py file
would raise ImportError (even if we said that was the intended
behaviour, there's nothing to enforce it).


Presumably the folks doing the splitting know what they are doing.  Any cPython 
auto-generated .missing.py files would be correct:

raise ImportError("tkinter was not compiled due to ...") .


Could the proposal execute the .missing.py file and then raise
ImportError? I could imagine that having problems of its own,
though...


Yeah, I don't think that's a good idea.

--
~Ethan~
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Chris Barker]

On Mon, Nov 28, 2016 at 1:26 PM, Paul Moore  wrote:

> > One "successful" use-case that would be impacted is the fallback import
> > idiom:
> >
> > try:
> > # this would do two full searches before getting the error
> > import BlahBlah
> > except ImportError:
> > import blahblah
>
> Under this proposal, the above idiom could potentially now fail.


higher on the thread, someone said that IMportError was not robust enough,
because it didn't give near,y as meaninful an error message as it might.

 > There's no guarantee that I can see that a .missing.py file

> would raise ImportError (even if we said that was the intended
> behaviour, there's nothing to enforce it).
>

there is nothing to enforce all sorts of things -- I dont hinkt it's so
wrong to have it in the spec that .missing.py fles NEED to raise an
ImportError

and they could give nice meaningful error messages that way without
breaking old code.

Could the proposal execute the .missing.py file and then raise
> ImportError? I could imagine that having problems of its own,
> though...


if the ImportError is raised by the surrounding code, then it would need a
protocol to get the nice error message -- raising an Exception is already a
protocol -- let's use that one.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Wolfgang Maier]

On 28.11.2016 22:26, Paul Moore wrote:

On 28 November 2016 at 21:11, Ethan Furman  wrote:

One "successful" use-case that would be impacted is the fallback import
idiom:

try:
# this would do two full searches before getting the error
import BlahBlah
except ImportError:
import blahblah


Under this proposal, the above idiom could potentially now fail. If
there's a BlahBlah.missing.py, then that will get executed rather than
an ImportError being raised, so the fallback wouldn't be executed.
This could actually be a serious issue for code that currently
protects against optional stdlib modules not being available like
this. There's no guarantee that I can see that a .missing.py file
would raise ImportError (even if we said that was the intended
behaviour, there's nothing to enforce it).

Could the proposal execute the .missing.py file and then raise
ImportError? I could imagine that having problems of its own,
though...



How about addressing both concerns by triggering the search for 
.missing.py only if an ImportError bubbles up uncaught (a bit similar to 
StopIteration nowadays)?


Wolfgang

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Paul Moore]

On 28 November 2016 at 21:11, Ethan Furman  wrote:
> One "successful" use-case that would be impacted is the fallback import
> idiom:
>
> try:
> # this would do two full searches before getting the error
> import BlahBlah
> except ImportError:
> import blahblah

Under this proposal, the above idiom could potentially now fail. If
there's a BlahBlah.missing.py, then that will get executed rather than
an ImportError being raised, so the fallback wouldn't be executed.
This could actually be a serious issue for code that currently
protects against optional stdlib modules not being available like
this. There's no guarantee that I can see that a .missing.py file
would raise ImportError (even if we said that was the intended
behaviour, there's nothing to enforce it).

Could the proposal execute the .missing.py file and then raise
ImportError? I could imagine that having problems of its own,
though...

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Ethan Furman]

On 11/28/2016 01:01 PM, Guido van Rossum wrote:

On Mon, Nov 28, 2016 at 12:51 PM, Ethan Furman wrote:

On 11/28/2016 05:28 AM, Tomas Orsava wrote:



Rendered PEP: https://fedora-python.github.io/pep-drafts/pep-A.html 




Overall +1, but using Guido's #2 option instead for handling
 *.missing.py files (searching all possible locations for the
module before falling back to the stdlib xxx.missing.py default).


Actually the .missing.py feature would be useful for other use
 cases, so it shouldn't be limited to the stdlib part of sys.path.
 (Also I'm withdrawing my idea of searching for it while searching
 for the original .py since that would burden successful imports
 with extra stat() calls.)


Absolutely.  The key point in your counter proposal is not failing at the first 
.missing.py file possible, but rather searching all possible locations first.

If we do the full search for the import first, then a full search for the 
.missing.py, and that ends up not hurting performance at all for successful 
imports -- well, that's just icing on the cake.  :)

One "successful" use-case that would be impacted is the fallback import idiom:

try:
# this would do two full searches before getting the error
import BlahBlah
except ImportError:
import blahblah

--
~Ethan~
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Ethan Furman]

On 11/28/2016 05:28 AM, Tomas Orsava wrote:


Rendered PEP: https://fedora-python.github.io/pep-drafts/pep-A.html


Overall +1, but using Guido's #2 option instead for handling *.missing.py files 
(searching all possible locations for the module before falling back to the 
stdlib xxx.missing.py default).

--
~Ethan~
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Terry Reedy]

On 11/28/2016 11:38 AM, Guido van Rossum wrote:

Overall I think this is a good idea. I have one hit:

It seems that there are two possible strategies for searching the
.missing.py file:

1. (Currently in the PEP) search it at the same time as the .py file
when walking along sys.path.
  - Pro: prevents confusion when the user accidentally has their own
matching file later in sys.path.
  - Con: prevents the user from installing a matching file intentionally
(e.g. a 3rd party version).

2. After exhausting sys.path, search it again just for .missing.py files
(or perhaps remember the location of the .missing.py file during the
first search but don't act immediately on it -- this has the same effect).
  - Pro: allows user to install their own version.


The Windows distribution. for instance, could have a mod.missing.py file 
for every non-Windows, unix-only module.  This would cut down on 'Why 
did this import fail?' questions on SO and python-list.  And without 
breaking anything. Even for non-beginners, it would save having to look 
up whether the an import failure is inherent on the platform or due to a 
typo.



  - Con: if the user has a matching file by accident, that file will be
imported, causing more confusion.

I personally would weigh these so as to prefer (2). The option of
installing your own version when the standard version doesn't exist
seems reasonable; there may be reasons that you can't or don't want to
install the distribution's version. I don't worry much about the danger
of accidental name conflicts (have you ever seen this?).


The accidental conflict reports I have seen were due to scripts in as 
in the same directory as the program, rather than modules in site-packages.

--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Nathaniel Smith]

On Nov 28, 2016 8:38 AM, "Guido van Rossum"  wrote:
>
> Overall I think this is a good idea. I have one hit:
>
> It seems that there are two possible strategies for searching the
.missing.py file:
>
> 1. (Currently in the PEP) search it at the same time as the .py file when
walking along sys.path.
>   - Pro: prevents confusion when the user accidentally has their own
matching file later in sys.path.
>   - Con: prevents the user from installing a matching file intentionally
(e.g. a 3rd party version).
>
> 2. After exhausting sys.path, search it again just for .missing.py files
(or perhaps remember the location of the .missing.py file during the first
search but don't act immediately on it -- this has the same effect).
>   - Pro: allows user to install their own version.
>   - Con: if the user has a matching file by accident, that file will be
imported, causing more confusion.
>
> I personally would weigh these so as to prefer (2). The option of
installing your own version when the standard version doesn't exist seems
reasonable; there may be reasons that you can't or don't want to install
the distribution's version. I don't worry much about the danger of
accidental name conflicts (have you ever seen this?).

I was going to make a similar comment, because it seems to me that it could
make sense for a redistributor to want to move some bits of the stdlib into
wheels, and this is most conveniently handled by letting bits of the stdlib
live in site-packages.

Also note that in Guido's option 2, we only incur the extra fstat calls if
the import would otherwise fail. In option 1, there are extra fstat calls
(and thus disk seeks etc.) adding some overhead to every import.

-n
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Alexandre Brault]

I would also prefer (2) for exactly the example given in this thread.
The Windows version of curses.missing.py could raise a
ModuleNotFoundError saying that curses is not available on Windows, but
a developer who wants to can install PDCurses to implement the stdlib
module.

I don't think the few cases where an stdlib package is both missing and
has a name collision with an incompatible module are enough to outweigh
the benefits of being able to install a third-party package to implement
a missing part of the stdlib

Alex


On 2016-11-28 11:38 AM, Guido van Rossum wrote:
> Overall I think this is a good idea. I have one hit:
>
> It seems that there are two possible strategies for searching the
> .missing.py file:
>
> 1. (Currently in the PEP) search it at the same time as the .py file
> when walking along sys.path.
>   - Pro: prevents confusion when the user accidentally has their own
> matching file later in sys.path.
>   - Con: prevents the user from installing a matching file
> intentionally (e.g. a 3rd party version).
>
> 2. After exhausting sys.path, search it again just for .missing.py
> files (or perhaps remember the location of the .missing.py file during
> the first search but don't act immediately on it -- this has the same
> effect).
>   - Pro: allows user to install their own version.
>   - Con: if the user has a matching file by accident, that file will
> be imported, causing more confusion.
>
> I personally would weigh these so as to prefer (2). The option of
> installing your own version when the standard version doesn't exist
> seems reasonable; there may be reasons that you can't or don't want to
> install the distribution's version. I don't worry much about the
> danger of accidental name conflicts (have you ever seen this?).
>
>
> --Guido
>
> On Mon, Nov 28, 2016 at 8:13 AM, Paul Moore  > wrote:
>
> On 28 November 2016 at 15:51, Tomas Orsava  > wrote:
> > I believe I may have found the Windows curses implementation,
> it's called
> > PDCurses [0], and this website [1] appears to be distributing it
> under the
> > name `curses`.
>
> My apologies, I should have included a pointer. That is indeed the
> distribution I was thinking of.
>
> > Could some Windows user please check if compiling Python with
> the current
> > reference implementation [2] of this PEP indeed generates a
> > `curses.missing.py ` file among the
> stdlib files? If so, we might consider
> > skipping the generation of the .missing.py file for the curses
> module on
> > Windows.
>
> I'll see if I can make some time to do the test. But as the change is
> to setup.py, and the Windows build uses Visual Studio project files to
> do the build, I expect that it won't generate missing.py files on
> Windows. In actual fact, that may be the simplest solution, to note
> that the build part of this change is restricted to Unix (non-Windows)
> platforms specifically. As there's no real concept of a "distribution
> version" of Python on Windows, it's probably not something that will
> be that important on that platform (and support for .missing.py files
> is there, it would just be necessary for distributors to manually
> create those files as needed).
>
> Paul
> ___
> Python-ideas mailing list
> Python-ideas@python.org 
> https://mail.python.org/mailman/listinfo/python-ideas
> 
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
>
>
>
>
> -- 
> --Guido van Rossum (python.org/~guido )
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Paul Moore]

On 28 November 2016 at 15:51, Tomas Orsava  wrote:
> I believe I may have found the Windows curses implementation, it's called
> PDCurses [0], and this website [1] appears to be distributing it under the
> name `curses`.

My apologies, I should have included a pointer. That is indeed the
distribution I was thinking of.

> Could some Windows user please check if compiling Python with the current
> reference implementation [2] of this PEP indeed generates a
> `curses.missing.py` file among the stdlib files? If so, we might consider
> skipping the generation of the .missing.py file for the curses module on
> Windows.

I'll see if I can make some time to do the test. But as the change is
to setup.py, and the Windows build uses Visual Studio project files to
do the build, I expect that it won't generate missing.py files on
Windows. In actual fact, that may be the simplest solution, to note
that the build part of this change is restricted to Unix (non-Windows)
platforms specifically. As there's no real concept of a "distribution
version" of Python on Windows, it's probably not something that will
be that important on that platform (and support for .missing.py files
is there, it would just be necessary for distributors to manually
create those files as needed).

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Tomas Orsava]

On 11/28/2016 03:32 PM, Paul Moore wrote:

On 28 November 2016 at 13:28, Tomas Orsava  wrote:

The ``.missing.py`` extension will be added to the end of the list, and
configured to be handled by ``SourceFileLoader``.  Thus, if a module is not
found in its proper location, the ``XYZ.missing.py`` file is found and
executed, and further locations are not searched.

Am I right to think that if a user had a file tkinter.missing.py in
the current directory, then they'd get that in preference to the
stdlib tkinter? Obviously this is no different from having a
tkinter.py file in that directory, so it's not like this is a major
problem, but it might be worth pointing out this minor
incompatibility.


Correct, both tkinter.py and tkinter.missing.py in the current directory 
will take precedence. I will note this in the backwards compatibility 
section.



Also, and possibly more of an issue, use of the ".missing.py" file
will mean that a user can't provide their own implementation of the
module later on sys.path. I don'rt know if this is a significant issue
on Unix platforms. On Windows, there is a 3rd party implementation of
the curses module which (as I understand it) can be user installed. If
Python included a curses.missing.py, that would no longer work.

Certainly these are only minor points, but worth considering.


I believe I may have found the Windows curses implementation, it's 
called PDCurses [0], and this website [1] appears to be distributing it 
under the name `curses`.


Could some Windows user please check if compiling Python with the 
current reference implementation [2] of this PEP indeed generates a 
`curses.missing.py` file among the stdlib files? If so, we might 
consider skipping the generation of the .missing.py file for the curses 
module on Windows.


[0] http://pdcurses.sourceforge.net/
[1] http://www.lfd.uci.edu/~gohlke/pythonlibs/#curses
[2] https://www.python.org/dev/peps/pep-0534/#reference-implementation


Thank you for the feedback!
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

[Paul Moore]

On 28 November 2016 at 13:28, Tomas Orsava  wrote:
> The ``.missing.py`` extension will be added to the end of the list, and
> configured to be handled by ``SourceFileLoader``.  Thus, if a module is not
> found in its proper location, the ``XYZ.missing.py`` file is found and
> executed, and further locations are not searched.

Am I right to think that if a user had a file tkinter.missing.py in
the current directory, then they'd get that in preference to the
stdlib tkinter? Obviously this is no different from having a
tkinter.py file in that directory, so it's not like this is a major
problem, but it might be worth pointing out this minor
incompatibility.

Also, and possibly more of an issue, use of the ".missing.py" file
will mean that a user can't provide their own implementation of the
module later on sys.path. I don'rt know if this is a significant issue
on Unix platforms. On Windows, there is a 3rd party implementation of
the curses module which (as I understand it) can be user installed. If
Python included a curses.missing.py, that would no longer work.

Certainly these are only minor points, but worth considering.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] PEP: Distributing a Subset of the Standard Library

[Tomas Orsava]

Hi!
We have written a draft PEP entitled "Distributing a Subset of the 
Standard Library" that aims to standardize and improve how Python 
handles omissions from its standard library. This is relevant both to 
Python itself as well as to Linux and other distributions that are 
packaging it, as they already separate out parts of the standard library 
into optionally installable packages.


Ideas leading up to this PEP were discussed on the python-dev mailing list:
https://mail.python.org/pipermail/python-dev/2016-July/145534.html

Rendered PEP: https://fedora-python.github.io/pep-drafts/pep-A.html



Abstract


Python is sometimes being distributed without its full standard library.
However, there is as of yet no standardized way of dealing with importing a
missing standard library module.  This PEP proposes a mechanism for 
identifying

which standard library modules are missing and puts forth a method of how
attempts to import a missing standard library module should be handled.


Motivation
==

There are several use cases for including only a subset of Python's standard
library.  However, there is so far no formal specification of how to 
properly

implement distribution of a subset of the standard library.  Namely, how to
safely handle attempts to import a missing *stdlib* module, and display an
informative error message.


CPython
---

When one of Python standard library modules (such as ``_sqlite3``) cannot be
compiled during a Python build because of missing dependencies (e.g. SQLite
header files), the module is simply skipped.

If you then install this compiled Python and use it to try to import one 
of the

missing modules, Python will go through the ``sys.path`` entries looking for
it.  It won't find it among the *stdlib* modules and thus it will 
continue onto
``site-packages`` and fail with a ModuleNotFoundError_ if it doesn't 
find it.


.. _ModuleNotFoundError:
https://docs.python.org/3.7/library/exceptions.html#ModuleNotFoundError

This can confuse users who may not understand why a cleanly built Python is
missing standard library modules.


Linux and other distributions
-

Many Linux and other distributions are already separating out parts of the
standard library to standalone packages.  Among the most commonly excluded
modules are the ``tkinter`` module, since it draws in a dependency on the
graphical environment, and the ``test`` package, as it only serves to test
Python internally and is about as big as the rest of the standard 
library put

together.

The methods of omission of these modules differ.  For example, Debian 
patches
the file ``Lib/tkinter/__init__.py`` to envelop the line ``import 
_tkinter`` in

a *try-except* block and upon encountering an ``ImportError`` it simply adds
the following to the error message: ``please install the python3-tk 
package``

[#debian-patch]_.  Fedora and other distributions simply don't include the
omitted modules, potentially leaving users baffled as to where to find them.


Specification
=

When, for any reason, a standard library module is not to be included 
with the
rest, a file with its name and the extension ``.missing.py`` shall be 
created

and placed in the directory the module itself would have occupied. This file
can contain any Python code, however, it *should* raise a 
ModuleNotFoundError_

with a helpful error message.

Currently, when Python tries to import a module ``XYZ``, the ``FileFinder``
path hook goes through the entries in ``sys.path``, and in each location 
looks
for a file whose name is ``XYZ`` with one of the valid suffixes (e.g. 
``.so``,

..., ``.py``, ..., ``.pyc``).  The suffixes are tried in order.  If none of
them are found, Python goes on to try the next directory in ``sys.path``.

The ``.missing.py`` extension will be added to the end of the list, and
configured to be handled by ``SourceFileLoader``.  Thus, if a module is not
found in its proper location, the ``XYZ.missing.py`` file is found and
executed, and further locations are not searched.

The CPython build system will be modified to generate ``.missing.py`` 
files for

optional modules that were not built.


Rationale
=

The mechanism of handling missing standard library modules through the 
use of

the ``.missing.py`` files was chosen due to its advantages both for CPython
itself and for Linux and other distributions that are packaging it.

The missing pieces of the standard library can be subsequently installed 
simply
by putting the module files in their appropriate location.  They will 
then take

precedence over the corresponding ``.missing.py`` files.  This makes
installation simple for Linux package managers.

This mechanism also solves the minor issue of importing a module from
``site-packages`` with the same name as a missing standard library module.
Now, Python will import the ``.missing.py`` file and won't