Re: [Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

[Barry Warsaw]

On Mar 1, 2019, at 02:41, Petr Viktorin  wrote:

> Currently, in RHEL/Fedora:
> - "sudo pip" installs to /usr/local/, RPM packages install to /usr/
> - with "-I", the interpreter does not look into /usr/local/.
> AFAIK, Debian/Ubuntu have something very similar, and were first to do it.

Debuntu’s pip installs to the user path by default, and you have to do 
something explicit to install into the system Python.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

[Petr Viktorin]

On 2/28/19 6:56 PM, Gregory P. Smith wrote:


On Wed, Feb 27, 2019 at 5:12 PM Toshio Kuratomi > wrote:



On Tue, Feb 26, 2019 at 2:07 PM Neil Schemenauer
mailto:nas-pyt...@arctrix.com>> wrote:

On 2019-02-26, Gregory P. Smith wrote:
 > On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw
mailto:ba...@python.org>> wrote:
 > For an OS distro provided interpreter, being able to restrict
its use to
 > only OS distro provided software would be ideal (so ideal
that people who
 > haven't learned the hard distro maintenance lessons may hate
me for it).

This idea has some definite problems.  I think enforcing it via
convention is about as much as would be good to do.  Anything more
and you make it hard for people who really need to use the vendor
provided interpreter from being able to do so.

Why might someone need to use the distro provided interpreter?

* Vendor provides some python modules in their system packages which
are not installable from pip (possibly even a proprietary extension
module, so not even buildable from source or copyable from the
system location) which the end user needs to use to do something to
their system.
* End user writes a python module which is a plugin to a system tool
which has to be installed into the system python to from which that
system tool runs.  The user then wants to write a script which uses
the system tool with the plugin in order to do something to their
system outside of the system tool (perhaps the system tool is
GUI-driven and the user wants to automate a part of it via the
python module).  They need their script to use the system python so
that they are using the same code as the system tool itself would use.

There's probably other scenarios where the benefits of locking the
user out of the system python outweigh the benefits but these are
the ones that I've run across lately.


Agreed.  The convention approach as someone said RHEL 8 has apparently 
done with an os distro reserved interpreter (yay) is likely good enough 
for most situations.


I'd go a /little/ further than that and suggest such an os distro 
reserved interpreter attempt to prevent installation of packages (ie: 
remove pip/ensurepip/distutils) via any other means than the OS package 
manager (rpms, debs).  Obviously that can't actually prevent someone 
from figuring out how to run getpip or manually installing trees of 
packages within its sys.path, but it acts as a deterrent suggesting that 
this interpreter is not intended for arbitrary software installation.


Currently, in RHEL/Fedora:
- "sudo pip" installs to /usr/local/, RPM packages install to /usr/
- with "-I", the interpreter does not look into /usr/local/.
AFAIK, Debian/Ubuntu have something very similar, and were first to do it.

What remains to tie this together is making "platform-python" always run 
with -I. This is PEP 432's "system-python" use case / design goal.
(The RHEL/Fedora platform-python was initially called system-python. We 
renamed to it so it doesn't clash with the PEP. It's the same use case, 
but we don't want to assign semantics to the name prematurely. 
Cutrrently, system-python is waiting for the larger-scale restructuring, 
and hopefully wider/longer discussion.)


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

[Gregory P. Smith]

On Wed, Feb 27, 2019 at 5:12 PM Toshio Kuratomi  wrote:

>
> On Tue, Feb 26, 2019 at 2:07 PM Neil Schemenauer 
> wrote:
>
>> On 2019-02-26, Gregory P. Smith wrote:
>> > On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw  wrote:
>> > For an OS distro provided interpreter, being able to restrict its use to
>> > only OS distro provided software would be ideal (so ideal that people
>> who
>> > haven't learned the hard distro maintenance lessons may hate me for it).
>>
>> This idea has some definite problems.  I think enforcing it via
> convention is about as much as would be good to do.  Anything more and you
> make it hard for people who really need to use the vendor provided
> interpreter from being able to do so.
>
> Why might someone need to use the distro provided interpreter?
>
> * Vendor provides some python modules in their system packages which are
> not installable from pip (possibly even a proprietary extension module, so
> not even buildable from source or copyable from the system location) which
> the end user needs to use to do something to their system.
> * End user writes a python module which is a plugin to a system tool which
> has to be installed into the system python to from which that system tool
> runs.  The user then wants to write a script which uses the system tool
> with the plugin in order to do something to their system outside of the
> system tool (perhaps the system tool is GUI-driven and the user wants to
> automate a part of it via the python module).  They need their script to
> use the system python so that they are using the same code as the system
> tool itself would use.
>
> There's probably other scenarios where the benefits of locking the user
> out of the system python outweigh the benefits but these are the ones that
> I've run across lately.
>
>
Agreed.  The convention approach as someone said RHEL 8 has apparently done
with an os distro reserved interpreter (yay) is likely good enough for most
situations.

I'd go a *little* further than that and suggest such an os distro reserved
interpreter attempt to prevent installation of packages (ie: remove
pip/ensurepip/distutils) via any other means than the OS package manager
(rpms, debs).  Obviously that can't actually prevent someone from figuring
out how to run getpip or manually installing trees of packages within its
sys.path, but it acts as a deterrent suggesting that this interpreter is
not intended for arbitrary software installation.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

[Toshio Kuratomi]

On Tue, Feb 26, 2019 at 2:07 PM Neil Schemenauer 
wrote:

> On 2019-02-26, Gregory P. Smith wrote:
> > On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw  wrote:
> > For an OS distro provided interpreter, being able to restrict its use to
> > only OS distro provided software would be ideal (so ideal that people who
> > haven't learned the hard distro maintenance lessons may hate me for it).
>
> This idea has some definite problems.  I think enforcing it via convention
is about as much as would be good to do.  Anything more and you make it
hard for people who really need to use the vendor provided interpreter from
being able to do so.

Why might someone need to use the distro provided interpreter?

* Vendor provides some python modules in their system packages which are
not installable from pip (possibly even a proprietary extension module, so
not even buildable from source or copyable from the system location) which
the end user needs to use to do something to their system.
* End user writes a python module which is a plugin to a system tool which
has to be installed into the system python to from which that system tool
runs.  The user then wants to write a script which uses the system tool
with the plugin in order to do something to their system outside of the
system tool (perhaps the system tool is GUI-driven and the user wants to
automate a part of it via the python module).  They need their script to
use the system python so that they are using the same code as the system
tool itself would use.

There's probably other scenarios where the benefits of locking the user out
of the system python outweigh the benefits but these are the ones that I've
run across lately.

-Toshio
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

[Steve Holden]

While these are interesting ideas, wouldn't it be better to leave this kind
of packaging to snap and similar utilities that bundle the language support
and libraries to allow simple isolated installation.

Kind regards
Steve Holden


On Tue, Feb 26, 2019 at 10:05 PM Neil Schemenauer 
wrote:

> On 2019-02-26, Gregory P. Smith wrote:
> > On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw  wrote:
> > For an OS distro provided interpreter, being able to restrict its use to
> > only OS distro provided software would be ideal (so ideal that people who
> > haven't learned the hard distro maintenance lessons may hate me for it).
>
> Interesting idea.  I remember when I was helping develop Debian
> packaging guides for Python software.   I had to fight with people
> to convince them that Debian packages should use
>
> #!/usr/bin/pythonX.Y
>
> rather than
>
> #!/usr/bin/env python
>
> The situtation is much better now but I still sometimes have
> packaged software fail because it picks up my version of
> /usr/local/bin/python.  I don't understand how people can believe
> grabbing /usr/local/bin/python is going to be a way to build a
> reliable system.
>
> > Such a restriction could be implemented within the interpreter itself.
> For
> > example: Say that only this set of fully qualified path whitelisted .py
> > files are allowed to invoke it, with no interactive, stdin, or command
> line
> > "-c" use allowed.
>
> I think this is related to an idea I was tinkering with on the
> weekend.  Why shouldn't we do more compile time linkage of Python
> packages?  At least, I think we give people the option to do it.
> Obviously you still need to also support run-time import search
> (interactive REPL, support __import__(unknown_at_compiletime)__).
>
> Here is the sketch of the idea (probably half-baked, as most of my
> ideas are):
>
> - add PYTHONPACKAGES envvar and -p options to 'python'
>
> - the argument for these options would be a colon separated list of
>   Python package archives (crates, bales, bundles?).  The -p option
>   could be a colon separated list or provided multiple times to
>   specify more packages.
>
> - the modules/packages contained in those archives become the
>   preferred bytecode code source when those names are imported.  We
>   look there first.  The crawling around behavor (dynamic import
>   based on sys.path) happens only if a module is not found and could
>   be turned off.
>
> - the linking of the modules could be computed when the code is
>   compiled and the package archive created, rather than when the
>   'import' statement gets executed.  This would provide a number of
>   advantages.  It would be faster.  Code analysis tools could
>   statically determine which modules imported code corresponds too.
>   E.g. if your code calls module.foo, assuming no monkey patching,
>   you know what code 'foo' actually is.
>
> - to get extra fancy, the package archives could be dynamic
>   link libraries containing "frozen modules" like this FB experiment:
>   https://github.com/python/cpython/pull/9320
>   That way, you avoid the unmarshal step and just execute the module
>   bytecode directly.  On startup, Python would dlopen all of the
>   package archives specified by PYTHONPACKAGES.  On init, it would
>   build an index of the package tree and it would have the memory
>   location for the code object for each module.
>
> That would seem like quite a useful thing.  For an application like
> Mercurial, they could build all the modules/packages required into a
> single package archive.  Or, there would be a small number of
> archives (one for standard Python library, one for everything else
> that Mercurial needs).
>
> Now that I write this, it sounds a lot like the debate between
> static linking and dynamic linking.  Golang does static linking and
> people seem to like the single executable distribution.
>
> Regards,
>
>   Neil
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

[Stephen J. Turnbull]

Barry Warsaw writes:

 >`/usr/bin/env python` is great for development and terrible for deployment. 

Developers of `six` and `2to3`, you mean?

Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

[Barry Warsaw]

On Feb 26, 2019, at 14:04, Neil Schemenauer  wrote:
> 
> Interesting idea.  I remember when I was helping develop Debian
> packaging guides for Python software.   I had to fight with people
> to convince them that Debian packages should use
> 
>#!/usr/bin/pythonX.Y
> 
> rather than
> 
>#!/usr/bin/env python

Indeed.  I used to fight that battle quite a bit, although at least in my 
circles that lesson has by now been learned.  `/usr/bin/env python` is great 
for development and terrible for deployment.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Compile-time resolution of packages [Was: Another update for PEP 394...]

[Neil Schemenauer]

On 2019-02-26, Gregory P. Smith wrote:
> On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw  wrote:
> For an OS distro provided interpreter, being able to restrict its use to
> only OS distro provided software would be ideal (so ideal that people who
> haven't learned the hard distro maintenance lessons may hate me for it).

Interesting idea.  I remember when I was helping develop Debian
packaging guides for Python software.   I had to fight with people
to convince them that Debian packages should use 

#!/usr/bin/pythonX.Y

rather than

#!/usr/bin/env python

The situtation is much better now but I still sometimes have
packaged software fail because it picks up my version of
/usr/local/bin/python.  I don't understand how people can believe
grabbing /usr/local/bin/python is going to be a way to build a
reliable system.

> Such a restriction could be implemented within the interpreter itself. For
> example: Say that only this set of fully qualified path whitelisted .py
> files are allowed to invoke it, with no interactive, stdin, or command line
> "-c" use allowed.

I think this is related to an idea I was tinkering with on the
weekend.  Why shouldn't we do more compile time linkage of Python
packages?  At least, I think we give people the option to do it.
Obviously you still need to also support run-time import search
(interactive REPL, support __import__(unknown_at_compiletime)__).

Here is the sketch of the idea (probably half-baked, as most of my
ideas are):

- add PYTHONPACKAGES envvar and -p options to 'python'

- the argument for these options would be a colon separated list of
  Python package archives (crates, bales, bundles?).  The -p option
  could be a colon separated list or provided multiple times to
  specify more packages.

- the modules/packages contained in those archives become the
  preferred bytecode code source when those names are imported.  We
  look there first.  The crawling around behavor (dynamic import
  based on sys.path) happens only if a module is not found and could
  be turned off.

- the linking of the modules could be computed when the code is
  compiled and the package archive created, rather than when the
  'import' statement gets executed.  This would provide a number of
  advantages.  It would be faster.  Code analysis tools could
  statically determine which modules imported code corresponds too.
  E.g. if your code calls module.foo, assuming no monkey patching,
  you know what code 'foo' actually is.

- to get extra fancy, the package archives could be dynamic
  link libraries containing "frozen modules" like this FB experiment:
  https://github.com/python/cpython/pull/9320
  That way, you avoid the unmarshal step and just execute the module
  bytecode directly.  On startup, Python would dlopen all of the
  package archives specified by PYTHONPACKAGES.  On init, it would
  build an index of the package tree and it would have the memory
  location for the code object for each module.

That would seem like quite a useful thing.  For an application like
Mercurial, they could build all the modules/packages required into a
single package archive.  Or, there would be a small number of
archives (one for standard Python library, one for everything else
that Mercurial needs).

Now that I write this, it sounds a lot like the debate between
static linking and dynamic linking.  Golang does static linking and
people seem to like the single executable distribution.

Regards,

  Neil
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com