Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Paul Moore]

On 22 January 2016 at 18:46, Nathaniel Smith  wrote:
>> I'd like to propose adding a special cased value to add to the installer
>> file
>> that will tell projects like pip that this particular installed thing is
>> being
>> managed by someone else, and we should keep our hands off of it. According
>> to
>> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I
>> think
>> since nobody has ever implemented it, we could expand that so that it so
>> you
>> can also have a special value, of "dpkg (system)" or maybe that's not
>> worth it
>> and we could just have "system" as a special value.
>
> I think we want more than just "system", because the same user could have
> some packages managed by dpkg and some by conda, both of which have their
> own dependency resolution mechanisms that are outside pip's and could get
> broken if pip removes stuff willy-nilly. And when pip errors out, you want
> to be able to explain to the user "this package is managed by conda, and
> using pip on it may break your conda setup..." versus "this package is
> managed by Debian, and using pip on it may break your Debian setup...".

Well, I would expect conda to be specifying "conda" in the INSTALLER
file. That's what the file is intended for, surely? To record what
tool installed the package?

Equally, there seems to me to be no reason dpkg couldn't use "dpkg",
and yum use "yum", etc. I just see Donald's suggestion of using
"system" as a simple flag saying "the OS" in a generic way for
distributions that don't (for some reason) *want* to name the specific
installer in the INSTALLER file. Well, that and the fact that they
could have done that for years now, so being able to say "just put
"system" in there and be done with it please!" is a nice easy message
to give to distributions.

Honestly, this seems to me to be a bit of a non-discussion.

My recollection of PEP 376 and the discussions around INSTALLER are
that it was simply a place where install tools could say "hey, it was
me that did this". It never got used largely because nobody seemed to
care about having that information. Now pip is starting to care, but
as far as I can see there are only 3 cases pip should care about:

1. The file says "pip". Do just as we do now, we control that package.
2. The file says something *other* than pip. We don't own the files,
take appropriate action. Warn, require an extra flag, whatever - we
can use what's in the file to say "files owned by FOO" but other than
that there's no reason we should behave differently depending on what
specific non-pip value we see.
3. There's no INSTALLER file. This is the awkward one, as we can't
(yet) say that this means the file isn't owned by pip. In a few years
maybe, but for now we have to assume it could be either of the above
cases.

So the message should be "if you want pip to be careful with your
files, put something (probably your installer name, but in practice
anything other than "pip") in INSTALLER".

Longer term, other tools might taking the same approach as pip, at
which point not using the same name as other tools becomes useful - at
that point, a generic term like "system" could be a bad choice...
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Donald Stufft]

> On Jan 22, 2016, at 3:38 PM, Paul Moore  wrote:
> 
> On 22 January 2016 at 18:46, Nathaniel Smith  wrote:
>>> I'd like to propose adding a special cased value to add to the installer
>>> file
>>> that will tell projects like pip that this particular installed thing is
>>> being
>>> managed by someone else, and we should keep our hands off of it. According
>>> to
>>> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I
>>> think
>>> since nobody has ever implemented it, we could expand that so that it so
>>> you
>>> can also have a special value, of "dpkg (system)" or maybe that's not
>>> worth it
>>> and we could just have "system" as a special value.
>> 
>> I think we want more than just "system", because the same user could have
>> some packages managed by dpkg and some by conda, both of which have their
>> own dependency resolution mechanisms that are outside pip's and could get
>> broken if pip removes stuff willy-nilly. And when pip errors out, you want
>> to be able to explain to the user "this package is managed by conda, and
>> using pip on it may break your conda setup..." versus "this package is
>> managed by Debian, and using pip on it may break your Debian setup...".
> 
> Well, I would expect conda to be specifying "conda" in the INSTALLER
> file. That's what the file is intended for, surely? To record what
> tool installed the package?
> 
> Equally, there seems to me to be no reason dpkg couldn't use "dpkg",
> and yum use "yum", etc. I just see Donald's suggestion of using
> "system" as a simple flag saying "the OS" in a generic way for
> distributions that don't (for some reason) *want* to name the specific
> installer in the INSTALLER file. Well, that and the fact that they
> could have done that for years now, so being able to say "just put
> "system" in there and be done with it please!" is a nice easy message
> to give to distributions.
> 
> Honestly, this seems to me to be a bit of a non-discussion.
> 
> My recollection of PEP 376 and the discussions around INSTALLER are
> that it was simply a place where install tools could say "hey, it was
> me that did this". It never got used largely because nobody seemed to
> care about having that information. Now pip is starting to care, but
> as far as I can see there are only 3 cases pip should care about:
> 
> 1. The file says "pip". Do just as we do now, we control that package.
> 2. The file says something *other* than pip. We don't own the files,
> take appropriate action. Warn, require an extra flag, whatever - we
> can use what's in the file to say "files owned by FOO" but other than
> that there's no reason we should behave differently depending on what
> specific non-pip value we see.
> 3. There's no INSTALLER file. This is the awkward one, as we can't
> (yet) say that this means the file isn't owned by pip. In a few years
> maybe, but for now we have to assume it could be either of the above
> cases.
> 
> So the message should be "if you want pip to be careful with your
> files, put something (probably your installer name, but in practice
> anything other than "pip") in INSTALLER".
> 
> Longer term, other tools might taking the same approach as pip, at
> which point not using the same name as other tools becomes useful - at
> that point, a generic term like "system" could be a bad choice...
> Paul

Hmm, in my head the three cases were more like:

1) The installed project is managed by a Python level tool which uses the 
Python metadata as it’s database. Thus if you’re this kind of tool too, then 
you can muck around here because things will be fine.
2) The installed project is managed by something that isn’t a Python level 
tool, and it has it’s own database and the .(egg|dist)-info files are not 
understood by it.
3) We don’t know what is managing the project.

The first would be things like, pip, easy_install, and distil. These can all 
easily interopt with each other because they’re all going to lay down .egg-info 
or .dist-info files and read those same files. The second is things like Conda, 
dpkg, yum, etc which are going to treat .egg-info or .dist-info files as just 
another opaque file that is included in it’s package that it has to lay down.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Paul Moore]

On 22 January 2016 at 20:45, Donald Stufft  wrote:
> Hmm, in my head the three cases were more like:
>
> 1) The installed project is managed by a Python level tool which uses the 
> Python metadata as it’s database. Thus if you’re this kind of tool too, then 
> you can muck around here because things will be fine.
> 2) The installed project is managed by something that isn’t a Python level 
> tool, and it has it’s own database and the .(egg|dist)-info files are not 
> understood by it.
> 3) We don’t know what is managing the project.
>
> The first would be things like, pip, easy_install, and distil. These can all 
> easily interopt with each other because they’re all going to lay down 
> .egg-info or .dist-info files and read those same files. The second is things 
> like Conda, dpkg, yum, etc which are going to treat .egg-info or .dist-info 
> files as just another opaque file that is included in it’s package that it 
> has to lay down.

Good point - maybe that's how it *should* have been. My recollection
of the discussions at the time was "put your tool name in here" and
nobody really thought about the possibility of more than one tool
co-operating. This was in the bad old days of "my tool is better than
yours" flamewars :-(

Switching to something like that model probably does need a PEP. But
INSTALLER doesn't seem the right concept for that - it's more like
wanting a file that defines how the project metadata is stored -
"Metadata 2.0" or "OS" or "Conda" or whatever.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Robert T. McGibbon]

On Jan 22, 2016 9:04 AM, "Lowekamp, Bradley (NIH/NLM/LHC) [C]" <
blowek...@mail.nih.gov> wrote:
>
> Hello,
>
> I noticed that libpython is missing from the lists of dependent
libraries. Also the “manylinux” Docker image has it’s Python versions
compiled with libpython static.
>
> Does this mean that we must do static linking against libpython?

This is a bug/imprecision in the PEP. Manylinux1 wheels *can* link against
libpython (the appropriate version for whatever python they're targeting),
and the latest version of the docker image uses a shared libpython now.

-Robert
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Donald Stufft]

> On Jan 22, 2016, at 1:46 PM, Nathaniel Smith  wrote:
> 
> On Jan 22, 2016 10:11 AM, "Donald Stufft"  > wrote:
> >
> > PEP 376 added a file to the .dist-info directory called "INSTALLER" which 
> > was
> > supposed to be:
> >
> > This option is the name of the tool used to invoke the installation.
> >
> > However, nothing has really ever implemented it and it's gone largely 
> > ignored
> > until just recently pip 8.0 started writing the INSTALLER file into the
> > metadata directories with a value of "pip".
> >
> > I'd like to propose adding a special cased value to add to the installer 
> > file
> > that will tell projects like pip that this particular installed thing is 
> > being
> > managed by someone else, and we should keep our hands off of it. According 
> > to
> > PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I 
> > think
> > since nobody has ever implemented it, we could expand that so that it so you
> > can also have a special value, of "dpkg (system)" or maybe that's not worth 
> > it
> > and we could just have "system" as a special value.
> 
> I think we want more than just "system", because the same user could have 
> some packages managed by dpkg and some by conda, both of which have their own 
> dependency resolution mechanisms that are outside pip's and could get broken 
> if pip removes stuff willy-nilly. And when pip errors out, you want to be 
> able to explain to the user "this package is managed by conda, and using pip 
> on it may break your conda setup..." versus "this package is managed by 
> Debian, and using pip on it may break your Debian setup...".
> 
> (Actually I'm not sure what the status these days is of mixing pip and conda 
> -- they've gotten somewhat better at handling it. Is the proposed behavior in 
> pip when it sees this flag something that distribution maintainers have asked 
> for? Are they present in this thread?)
> 
> 

Yea, that’s why I thought about dpkg (system) or system(Debian) or something. 
The main reason I can think of for preferring “system” is if we don’t want to 
change the valid characters for a value in this file. Then you can have 
system(Debian) and system(Conda) and everything will work just fine.
> > The benefit of doing this, is that with a special value in that file that 
> > says
> > "this file belongs to the OS", then pip could start looking for that file 
> > and
> > require a --force flag before it modifies any files belonging to that 
> > project.
> > Then distributors like Debian, Fedora, etc could simply write out the 
> > INSTALLER
> > file with the correct value, and pip would start to respect their files by
> > default.
> 
> I'd like a little more clarity on exactly what circumstances justify setting 
> this flag. If I write a new python package manager, then should I set this 
> flag on all my packages because I don't trust anyone else to get things 
> right? :-)
> 
> Maybe the relevant thing is what I said above, that there is some system 
> tracking these files that is not using the dist-info directory as its 
> source-of-truth about what's installed, dependencies, etc.
> 
> -n
> 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Donald Stufft]

> On Jan 22, 2016, at 2:09 PM, Robert Collins  wrote:
> 
> I think requiring a force flag to replace files installed by another
> packaging tool is an unbreakme option as things stand, and so I'm very
> concerned about the resulting user experience.

I don't think the current behavior makes *any* sense and represents nothing but
a footgun where you end up with two systems fighting over what the content of a
particular file is.

Currently there are two sort of scenarios where something like this comes up:

Debuntu
---

Current behavior is we'll uninstall the file sitting at
/usr/lib/pythonX.Y/dist-packages/* and then we'll go ahead and install our own
files into /usr/local/lib/pythonX.Y/dist-packages/*. Then, the next time that
package gets an update the files are put back into place by dpkg into
/usr/lib/pythonX.Y/dist-packages/* and we're finally at a consistent state
where both package managers know the state of their respective systems.

In the "you need a flag" future world, we would just skip uninstalling the
files from /usr/lib/pythonX.Y/dist-packages/* and just install our own files
into /usr/local/lib/pythonX.Y/dist-packages. Thus we never touch anything that
isn't owned by us and Debian's internal systems stay consistent. If someone did
only ``pip uninstall requests`` and that was located in the OS files, then we
would refuse to uninstall it, directing them to the platform tooling instead.


Everyone Else
-

The /usr/lib and /usr/local/lib split is Debian specific (though at some point
I want to bring it to upstream Python) so on other systems you have a single
location, typically in /usr/lib/.../site-packages where both the OS and pip
will be trying to modify files. Right now this is pure chaos, if
python-requests is installed by the system, pip will come along and uninstall
it then upgrade requests, then the next time an update is released, the system
tooling will then come along and write all over the files that pip dropped
into place, possibly corrupting things depending on how their patching works.
Even for just ``pip uninstall foo``, it's really only uninstall until the
next time that package gets updated.

In the "you need a flag future", if python-requests is installed by the OS and
it's in the same location that *we* want to write a file to, then we'll error
out and require a --force flag. If it isn't already installed and nothing is
in our way, then we'll go ahead and install to that location. This isn't as
nice as the Debuntu world, but nothing will be without the patch that Debian
has.


Essentially though, ``pip install -U anything-installed-by-the-os`` is at best
a no-op that deletes files for no good reason (Debuntu) or it's a field full of
footguns where you're going to have two independent systems shitting all over
each others files trying to do what they think the user wants.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Paul Moore]

On 22 January 2016 at 18:10, Donald Stufft  wrote:
> The benefit of doing this, is that with a special value in that file that says
> "this file belongs to the OS", then pip could start looking for that file and
> require a --force flag before it modifies any files belonging to that project.
> Then distributors like Debian, Fedora, etc could simply write out the 
> INSTALLER
> file with the correct value, and pip would start to respect their files by
> default.
>
> Thoughts? Does this sound reasonable to everyone? Do we think it needs a new
> PEP or can we just amend PEP 376 to add this extra bit of information?

Using "system" as the name of the "installer" sounds like it conforms
pretty well to PEP 376 as it stands.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Ian Cordasco]

On Fri, Jan 22, 2016 at 12:10 PM, Donald Stufft  wrote:
> PEP 376 added a file to the .dist-info directory called "INSTALLER" which was
> supposed to be:
>
> This option is the name of the tool used to invoke the installation.
>
> However, nothing has really ever implemented it and it's gone largely ignored
> until just recently pip 8.0 started writing the INSTALLER file into the
> metadata directories with a value of "pip".
>
> I'd like to propose adding a special cased value to add to the installer file
> that will tell projects like pip that this particular installed thing is being
> managed by someone else, and we should keep our hands off of it. According to
> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I think
> since nobody has ever implemented it, we could expand that so that it so you
> can also have a special value, of "dpkg (system)" or maybe that's not worth it
> and we could just have "system" as a special value.

Why not use something like JSON to allow us to have a little more
information about the installer, e.g.,

{"name": "pip", "system": true, ...}

> The benefit of doing this, is that with a special value in that file that says
> "this file belongs to the OS", then pip could start looking for that file and
> require a --force flag before it modifies any files belonging to that project.
> Then distributors like Debian, Fedora, etc could simply write out the 
> INSTALLER
> file with the correct value, and pip would start to respect their files by
> default.
>
> Thoughts? Does this sound reasonable to everyone? Do we think it needs a new
> PEP or can we just amend PEP 376 to add this extra bit of information?

I think amending the PEP makes sense.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Nathaniel Smith]

On Jan 22, 2016 10:11 AM, "Donald Stufft"  wrote:
>
> PEP 376 added a file to the .dist-info directory called "INSTALLER" which
was
> supposed to be:
>
> This option is the name of the tool used to invoke the installation.
>
> However, nothing has really ever implemented it and it's gone largely
ignored
> until just recently pip 8.0 started writing the INSTALLER file into the
> metadata directories with a value of "pip".
>
> I'd like to propose adding a special cased value to add to the installer
file
> that will tell projects like pip that this particular installed thing is
being
> managed by someone else, and we should keep our hands off of it.
According to
> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I
think
> since nobody has ever implemented it, we could expand that so that it so
you
> can also have a special value, of "dpkg (system)" or maybe that's not
worth it
> and we could just have "system" as a special value.

I think we want more than just "system", because the same user could have
some packages managed by dpkg and some by conda, both of which have their
own dependency resolution mechanisms that are outside pip's and could get
broken if pip removes stuff willy-nilly. And when pip errors out, you want
to be able to explain to the user "this package is managed by conda, and
using pip on it may break your conda setup..." versus "this package is
managed by Debian, and using pip on it may break your Debian setup...".

(Actually I'm not sure what the status these days is of mixing pip and
conda -- they've gotten somewhat better at handling it. Is the proposed
behavior in pip when it sees this flag something that distribution
maintainers have asked for? Are they present in this thread?)

> The benefit of doing this, is that with a special value in that file that
says
> "this file belongs to the OS", then pip could start looking for that file
and
> require a --force flag before it modifies any files belonging to that
project.
> Then distributors like Debian, Fedora, etc could simply write out the
INSTALLER
> file with the correct value, and pip would start to respect their files by
> default.

I'd like a little more clarity on exactly what circumstances justify
setting this flag. If I write a new python package manager, then should I
set this flag on all my packages because I don't trust anyone else to get
things right? :-)

Maybe the relevant thing is what I said above, that there is some system
tracking these files that is not using the dist-info directory as its
source-of-truth about what's installed, dependencies, etc.

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP 376, the INSTALLER file, and system packages

[Donald Stufft]

PEP 376 added a file to the .dist-info directory called "INSTALLER" which was
supposed to be:

This option is the name of the tool used to invoke the installation.

However, nothing has really ever implemented it and it's gone largely ignored
until just recently pip 8.0 started writing the INSTALLER file into the
metadata directories with a value of "pip".

I'd like to propose adding a special cased value to add to the installer file
that will tell projects like pip that this particular installed thing is being
managed by someone else, and we should keep our hands off of it. According to
PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I think
since nobody has ever implemented it, we could expand that so that it so you
can also have a special value, of "dpkg (system)" or maybe that's not worth it
and we could just have "system" as a special value.

The benefit of doing this, is that with a special value in that file that says
"this file belongs to the OS", then pip could start looking for that file and
require a --force flag before it modifies any files belonging to that project.
Then distributors like Debian, Fedora, etc could simply write out the INSTALLER
file with the correct value, and pip would start to respect their files by
default.

Thoughts? Does this sound reasonable to everyone? Do we think it needs a new
PEP or can we just amend PEP 376 to add this extra bit of information?

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Fred Drake]

On Fri, Jan 22, 2016 at 1:10 PM, Donald Stufft  wrote:
> Does this sound reasonable to everyone? Do we think it needs a new
> PEP or can we just amend PEP 376 to add this extra bit of information?

Identifying something special like "system" doesn't seem bad, and
conforms (assuming PEP 376 really meant to include "+" at the end of
the regular expression).

If tools find the INSTALLER file and it identifies some other tool,
though, it shouldn't matter if there's a special "system" value.
Debian systems could use "dpkg" or "apt", RPM-based systems could use
"rpm" or "yum", and it sounds like the new pip would be just as happy
to do the right thing.

I don't think a new or updated PEP is needed.  I'm glad to see this
being addressed.  Thanks for all your hard work, Donald!


  -Fred

-- 
Fred L. Drake, Jr.
"A storm broke loose in my mind."  --Albert Einstein
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Robert Collins]

On 23 January 2016 at 07:10, Donald Stufft  wrote:
> PEP 376 added a file to the .dist-info directory called "INSTALLER" which was
> supposed to be:
>
> This option is the name of the tool used to invoke the installation.
>
> However, nothing has really ever implemented it and it's gone largely ignored
> until just recently pip 8.0 started writing the INSTALLER file into the
> metadata directories with a value of "pip".
>
> I'd like to propose adding a special cased value to add to the installer file
> that will tell projects like pip that this particular installed thing is being
> managed by someone else, and we should keep our hands off of it. According to
> PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I think
> since nobody has ever implemented it, we could expand that so that it so you
> can also have a special value, of "dpkg (system)" or maybe that's not worth it
> and we could just have "system" as a special value.
>
> The benefit of doing this, is that with a special value in that file that says
> "this file belongs to the OS", then pip could start looking for that file and
> require a --force flag before it modifies any files belonging to that project.
> Then distributors like Debian, Fedora, etc could simply write out the 
> INSTALLER
> file with the correct value, and pip would start to respect their files by
> default.
>
> Thoughts? Does this sound reasonable to everyone? Do we think it needs a new
> PEP or can we just amend PEP 376 to add this extra bit of information?

I think asking other distros to export the installing information to
us is a good thing.

I think requiring a force flag to replace files installed by another
packaging tool is an unbreakme option as things stand, and so I'm very
concerned about the resulting user experience.

If someone runs 'sudo pip install -U pip' they are already signalling
precisely what they want, and as a user I very much doubt that:

"Refusing to uninstall pip because it was not installed by pip"

will make even 5% of them pause longer than required to look up the
--force-alternative-installer flag (or whatever it is) and pass it in.

The only reason we're touching these other installer installed files
is because there is no location on the Python and exec paths where we
can install the new pip without overwriting the one dpkg (or conda or
whatever) installed. If there was such a place we could just write
(and upgrade things) there and not have to deal with shadowing, or
ever uninstalling dpkg etc installed files. In the absence of such a
place, removing the other-install-system's installed files is the
right thing to do when a user asks us to, without needing an
additional option.

We could add an option folk could set to turn *on* such a safety belt
- but I don't know who it would /really/ be helping. (I'm fairly sure
I know the places folk think it would help, I just don't agree that it
would :)).

Having a interface to the system package manager as has been
previously mooted - and I'm going to get back to that - might help a
little, but uninstalls are quite different to installs. Uninstalls get
blocked by things like 'app-foo requires requests', and I would be
super suprised (and delighted!) if we were able to override that when
upgrading requests via pip :)

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[M.-A. Lemburg]

On 22.01.2016 11:03, Nathaniel Smith wrote:
> On Fri, Jan 22, 2016 at 1:33 AM, M.-A. Lemburg  wrote:
>> On 21.01.2016 20:05, Matthew Brett wrote:
>>> Hi,
>>>
>>> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg  wrote:
 On 21.01.2016 10:31, Nick Coghlan wrote:
> On 21 January 2016 at 19:03, M.-A. Lemburg  wrote:
>> By using the version based approach, we'd not run into this
>> problem and gain a lot more.
>
> I think it's better to start with a small core that we *know* works,
> then expand later, rather than trying to make the first iteration too
> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
> end), so "manylinux2" may simply have *more* libraries defined, rather
> than newer ones.

 My argument is that the file based approach taken by the PEP
 is too limiting to actually make things work for a large
 set of Python packages.

 It will basically only work for packages that do not interface
 to other external libraries (except for the few cases listed in
 the PEP, e.g. X11, GL, which aren't always installed or
 available either).

 IMO, testing the versions of a set of libraries is a safer
 approach. It's perfectly fine to have a few dependencies
 not work in a module because an optional system package is not
 installed, e.g. say a package comes with UIs written in
 Qt and one in GTK.
>>>
>>> Please forgive my slowness, but I don't understand exactly what you
>>> mean.  Can you give a specific example?
>>>
>>> Say my package depends on libpng.
>>>
>>> Call the machine I'm installing on the client machine.
>>>
>>> Are you saying that, when I build a wheel, I should specify to the
>>> wheel what versions of libpng I can tolerate on the the client
>>> machine, and if if the client does have a compatible version, then pip
>>> should raise an error, perhaps with a useful message about how to get
>>> libpng?
>>>
>>> If you do mean that, how do you want the PEP changed?
>>
>> I already posted a change proposal earlier on in the thread.
>> I'll repeat it here (with a minor enhancements):
> 
> Okay, I think I get it now. I'll try to repeat back to summarize and
> see if I have understood your proposal correctly:
> 
> In the PEP 513 "manylinux1" approach, when users do 'pip install foo',
> then one of three things happens:
> 1) they get a working foo and are immediately good-to-go, or
> 2) pip says "I'm sorry, there's no compatible wheel", or
> 3) something else happens, in which case this is a bug, and the spec
> provides some framework to help us determine whether this is a bug in
> the wheel, a bug in pip, or a bug in the spec.
> 
> In your approach, users do 'pip install foo', and then pip installs
> the wheel, and then when they try to use the wheel they get an error
> message from the dynamic linker about missing libraries, and then the
> user has to read the docs or squint at these error messages in order
> to figure out what set of apt-get / yum / pacman / ... commands they
> need to run in order to make foo work. (And possibly there is no such
> combination of commands that will actually work, because e.g. the
> wheel was linked against Debian's version of libbar.so.7 and Fedora's
> version of libbar.so.7 turns out to have an incompatible ABI, or
> Fedora simply doesn't provide a libbar.so.7 package at all.)

pip could be made to check the wheel for missing library
dependencies in order to provide help with cases where
additional packages are needed, but overall, yes, that's
the way it should work, IMO.

It's better to have wheels than not to have them, since
installing an additional system package is by far easier
than trying to compile packages from source (this will
usually also require additional -dev packages to be
installed).

>>  * no lock-out of package authors who would like to push
>>wheel files for their packages to PyPI, but happen to
>>use libraries not in the predefined list of the original
>>draft PEP
>
> https://mail.python.org/pipermail/distutils-sig/2016-January/028050.html

Embedding additional libraries in the wheels files to overcome
deficiencies in the PEP design simply doesn't feel right
to me.

People who rely on Linux distributions want to continue
to do so and get regular updates for system packages from
their system vendor. Having wheel files override these
system packages by including libs directly in the wheel
silently breaks this expectation, potentially opening
up the installations for security holes, difficult to
track bugs and possible version conflicts with already
loaded versions of the shared libs.

IMO, that's much worse than having to install additional
system packages to make a Python wheel work.

The embedding approach also creates licensing problems,
since those libs may be under different licenses than the
package itself. And of course, it increases the size of the
wheel files, 

Re: [Distutils] draft PEP: manylinux1

[Paul Moore]

On 22 January 2016 at 07:04, Matthew Brett  wrote:
>> That's an interesting idea, but I personally don't see the manylinux1 list
>> as particularly
>> "scientific". If anything, I'd call it "minimal".
>
> Yes, I agree, I don't think 'linux-sciabi1" would differentiate this
> from other ways of building wheels.  For example, I can't see why this
> wouldn't be a perfectly reasonable way to proceed for someone doing
> audio or video.   The difference that "manylinux" was designed to
> capture is the idea of having a single set of wheels for many versions
> of Linux, rather than wheels specific to particular distributions or
> packaged versions of external libraries.

Experience with Christoph Gohlke's binary distributions on Windows
suggests that a significant majority of non-scientific uses are
perfectly well served by the sort of package list that scientific
users would need. And I suspect that not all Enthought/Anaconda users
are scientists, either.

So I'd rather that the tag was based on capability rather than
community / intended use. On that basis, "linux-minimal1" sounds fine
to me.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Nathaniel Smith]

On Fri, Jan 22, 2016 at 1:33 AM, M.-A. Lemburg  wrote:
> On 21.01.2016 20:05, Matthew Brett wrote:
>> Hi,
>>
>> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg  wrote:
>>> On 21.01.2016 10:31, Nick Coghlan wrote:
 On 21 January 2016 at 19:03, M.-A. Lemburg  wrote:
> By using the version based approach, we'd not run into this
> problem and gain a lot more.

 I think it's better to start with a small core that we *know* works,
 then expand later, rather than trying to make the first iteration too
 wide. The "manylinux1" tag itself is versioned (hence the "1" at the
 end), so "manylinux2" may simply have *more* libraries defined, rather
 than newer ones.
>>>
>>> My argument is that the file based approach taken by the PEP
>>> is too limiting to actually make things work for a large
>>> set of Python packages.
>>>
>>> It will basically only work for packages that do not interface
>>> to other external libraries (except for the few cases listed in
>>> the PEP, e.g. X11, GL, which aren't always installed or
>>> available either).
>>>
>>> IMO, testing the versions of a set of libraries is a safer
>>> approach. It's perfectly fine to have a few dependencies
>>> not work in a module because an optional system package is not
>>> installed, e.g. say a package comes with UIs written in
>>> Qt and one in GTK.
>>
>> Please forgive my slowness, but I don't understand exactly what you
>> mean.  Can you give a specific example?
>>
>> Say my package depends on libpng.
>>
>> Call the machine I'm installing on the client machine.
>>
>> Are you saying that, when I build a wheel, I should specify to the
>> wheel what versions of libpng I can tolerate on the the client
>> machine, and if if the client does have a compatible version, then pip
>> should raise an error, perhaps with a useful message about how to get
>> libpng?
>>
>> If you do mean that, how do you want the PEP changed?
>
> I already posted a change proposal earlier on in the thread.
> I'll repeat it here (with a minor enhancements):

Okay, I think I get it now. I'll try to repeat back to summarize and
see if I have understood your proposal correctly:

In the PEP 513 "manylinux1" approach, when users do 'pip install foo',
then one of three things happens:
1) they get a working foo and are immediately good-to-go, or
2) pip says "I'm sorry, there's no compatible wheel", or
3) something else happens, in which case this is a bug, and the spec
provides some framework to help us determine whether this is a bug in
the wheel, a bug in pip, or a bug in the spec.

In your approach, users do 'pip install foo', and then pip installs
the wheel, and then when they try to use the wheel they get an error
message from the dynamic linker about missing libraries, and then the
user has to read the docs or squint at these error messages in order
to figure out what set of apt-get / yum / pacman / ... commands they
need to run in order to make foo work. (And possibly there is no such
combination of commands that will actually work, because e.g. the
wheel was linked against Debian's version of libbar.so.7 and Fedora's
version of libbar.so.7 turns out to have an incompatible ABI, or
Fedora simply doesn't provide a libbar.so.7 package at all.)

I won't express any opinion on your alternative PEP with its own
platform tag without reading it, but we're not going to change PEP 513
to work this way.

>  * no lock-out of package authors who would like to push
>wheel files for their packages to PyPI, but happen to
>use libraries not in the predefined list of the original
>draft PEP

https://mail.python.org/pipermail/distutils-sig/2016-January/028050.html

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[M.-A. Lemburg]

On 21.01.2016 20:05, Matthew Brett wrote:
> Hi,
> 
> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg  wrote:
>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>> On 21 January 2016 at 19:03, M.-A. Lemburg  wrote:
 By using the version based approach, we'd not run into this
 problem and gain a lot more.
>>>
>>> I think it's better to start with a small core that we *know* works,
>>> then expand later, rather than trying to make the first iteration too
>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>> than newer ones.
>>
>> My argument is that the file based approach taken by the PEP
>> is too limiting to actually make things work for a large
>> set of Python packages.
>>
>> It will basically only work for packages that do not interface
>> to other external libraries (except for the few cases listed in
>> the PEP, e.g. X11, GL, which aren't always installed or
>> available either).
>>
>> IMO, testing the versions of a set of libraries is a safer
>> approach. It's perfectly fine to have a few dependencies
>> not work in a module because an optional system package is not
>> installed, e.g. say a package comes with UIs written in
>> Qt and one in GTK.
> 
> Please forgive my slowness, but I don't understand exactly what you
> mean.  Can you give a specific example?
> 
> Say my package depends on libpng.
> 
> Call the machine I'm installing on the client machine.
> 
> Are you saying that, when I build a wheel, I should specify to the
> wheel what versions of libpng I can tolerate on the the client
> machine, and if if the client does have a compatible version, then pip
> should raise an error, perhaps with a useful message about how to get
> libpng?
> 
> If you do mean that, how do you want the PEP changed?

I already posted a change proposal earlier on in the thread.
I'll repeat it here (with a minor enhancements):

"""
The ``manylinux1`` policy
=

For these reasons, we define a set of library versions that
are supported by a wide range of Linux distributions. We therefore
pick library versions which have been around for at least 5 years.

When using these external libraries, Python wheels should
only depend on library versions listed in the section below.

Python wheels are free to depend on additional libraries
not included in this set, however, care should be taken
that these additional libraries do not depend on later
versions of the listed libraries, e.g. OpenSSL libraries
compiled against the C library versions listed below.

The ``manylinux1`` policy thus encompasses a standard for which
versions of these external shared libraries a wheel may depend on,
and the maximum depended-upon symbol versions therein.

Future versions of the manylinux policy may include more
libraries, or move on to later versions.

The permitted external shared libraries versions for
``manylinux1``are: ::

libgcc_s.so.1
libstdc++.so.6
... only include the basic set of libs, no GUI or curses ...

"""

The idea is to not pin down the set of usable external libraries,
but instead pin down a set of versions for the most important
libraries wheels will depend on and then let the wheels use
other external libraries as necessary without version checks.

In more details:

If you want a wheel to run on many Linux distributions, you have
to make sure that the basic lib C and a few other utility
libraries are available and compatible with the ones you used
to build the wheel.

This can be addressed by defining a set of important libraries
and corresponding versions. You do not have to limit the overall
set of usable libraries for this, since less commonly used
libraries will usually have to be installed separately anyway.

For example, if a package needs a specific version of libpng,
the package author can document this and the user can then make
sure to install that particular version.

The PEP should only be concerned with the basic set of
libraries you typically need for a wheel, not any of the
less common ones. The X11 libs for example do not have to
be version pinned for the manylinux tag, since they are not
essential for the vast majority of Python packages (and here
I'm talking about the thousands of packages on PyPI, not the
few hundred mentioned earlier in the thread, which are
covered by Anaconda and Canopy).

By defining "manylinux1" in such a way you get:

 * broad compatibility of wheel files on Linux

 * full flexibility of wheels interfacing or wrapping
   to other external libraries not covered in the PEP

 * no lock-out of package authors who would like to push
   wheel files for their packages to PyPI, but happen to
   use libraries not in the predefined list of the original
   draft PEP

I left out the other details I mentioned (symbol versioning
and dealing with different architectures) to focus on
pinning libraries vs. pinning versions for now. Later 

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Nathaniel Smith]

On Fri, Jan 22, 2016 at 11:28 AM, Donald Stufft  wrote:

>
> On Jan 22, 2016, at 1:46 PM, Nathaniel Smith  wrote:
>
> On Jan 22, 2016 10:11 AM, "Donald Stufft"  wrote:
> >
> > PEP 376 added a file to the .dist-info directory called "INSTALLER"
> which was
> > supposed to be:
> >
> > This option is the name of the tool used to invoke the installation.
> >
> > However, nothing has really ever implemented it and it's gone largely
> ignored
> > until just recently pip 8.0 started writing the INSTALLER file into the
> > metadata directories with a value of "pip".
> >
> > I'd like to propose adding a special cased value to add to the installer
> file
> > that will tell projects like pip that this particular installed thing is
> being
> > managed by someone else, and we should keep our hands off of it.
> According to
> > PEP 376 the supported values for this file are r"[a-z0-9_-.]", however I
> think
> > since nobody has ever implemented it, we could expand that so that it so
> you
> > can also have a special value, of "dpkg (system)" or maybe that's not
> worth it
> > and we could just have "system" as a special value.
>
> I think we want more than just "system", because the same user could have
> some packages managed by dpkg and some by conda, both of which have their
> own dependency resolution mechanisms that are outside pip's and could get
> broken if pip removes stuff willy-nilly. And when pip errors out, you want
> to be able to explain to the user "this package is managed by conda, and
> using pip on it may break your conda setup..." versus "this package is
> managed by Debian, and using pip on it may break your Debian setup...".
>
> (Actually I'm not sure what the status these days is of mixing pip and
> conda -- they've gotten somewhat better at handling it. Is the proposed
> behavior in pip when it sees this flag something that distribution
> maintainers have asked for? Are they present in this thread?)
>
>
> Yea, that’s why I thought about dpkg (system) or system(Debian) or
> something. The main reason I can think of for preferring “system” is if we
> don’t want to change the valid characters for a value in this file. Then
> you can have system(Debian) and system(Conda) and everything will work just
> fine.
>

Maybe to simplify the discussion we should just forget about INSTALLER,
leave it in peace to be what the PEP says ("fyi this is the last program to
touch this, in case anyone cares"), and add a new file with whatever
syntax/semantics make sense. Filenames are cheap and plentiful, and no
reason to complicate this discussion with hypothetical backcompat worries.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org 
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Nick Coghlan]

On 22 January 2016 at 17:04, Matthew Brett  wrote:
> Hi,
> On Thu, Jan 21, 2016 at 7:45 PM, Robert T. McGibbon  
> wrote:
>> On Thu, Jan 21, 2016 at 7:32 PM, Nick Coghlan  wrote:
>>> However, it does suggest a possible alternative approach to naming
>>> these compatibility subsets: what if the name of this particular
>>> platform compatibility tag was something like "linux-sciabi1", rather
>>> than "manylinux1"?
>>
>> That's an interesting idea, but I personally don't see the manylinux1 list
>> as particularly
>> "scientific". If anything, I'd call it "minimal".
>
> Yes, I agree, I don't think 'linux-sciabi1" would differentiate this
> from other ways of building wheels.  For example, I can't see why this
> wouldn't be a perfectly reasonable way to proceed for someone doing
> audio or video.   The difference that "manylinux" was designed to
> capture is the idea of having a single set of wheels for many versions
> of Linux, rather than wheels specific to particular distributions or
> packaged versions of external libraries.

Yeah, it was just an idea to potentially address MAL's concerns
regarding scope. However, I think the other replies to the thread have
adequately addressed that, and we can continue deferring the question
of scope increases to manylinux2 after seeing how far the current list
and "auditwheel repair" can get us.

The PEP should also be explicit that this does reintroduce the
bundling problem that distro unbundling policies were designed to
address, but:

1. In these days of automated continuous intregration & deployment
pipelines, publishing new versions and updating dependencies is easier
than it was when those policies were defined
2. Folks remain free to use "--no-binary" if they want to force local
builds rather than using pre-built wheel files
3. The popularity of container based deployment and "immutable
infrastructure" models involve substantial bundling at the application
layer anyway
4. This PEP doesn't rule out the idea of offering more targeted
binaries for particular Linux distributions

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Olivier Grisel]

2016-01-22 3:47 GMT+01:00 Chris Barker - NOAA Federal :
>
> Maybe the community will spring forth and do that -- I'm skeptical because I
> tried to to that for years for OS-X and it was just too much to do. And the
> infrastructure was there.
>
> Before pip and wheel there were mpkgs on OS-X, and repo's of toms for Linux
> years before that -- but always the result of a couple people's heroic
> efforts.
>
> Maybe the infrastructure has improved, and the community grown enough, that
> this will all work. We'll see.

I think the infrastructure has improved. For instance I invested some
time and effort to provide a template configuration to build C/C++
compiled extensions for windows wheels on top of the free AppVeyor.com
CI platform [1].

Since then this build configuration has been integrated in the python
packaging documentation [2] and I had the opportunity to present that
work at PyCon [3] (and PyCon FR) last year. Now the community of
project maintainers has picked it up. I can count more than 300
projects using this build setup on github. I have very little work to
do to help them maintain that infra-strucuture nowadays. Even the
configuration upgrade to make it work with MSVC 2015 / Python 3.5 was
contributed to my repo before I could find the time to investigate the
issue my-self.

My point is that once we have clearly defined best-practices for
packaging and convenient tools to build the packages automatically and
test that they work as expected (e.g. free hosted CI that support
running an old centos-based docker container), I am rather confident
that the community will do the work.

It's mostly a matter of providing good tools and education resources
(documentation and example configuration templates).

[1] https://github.com/ogrisel/python-appveyor-demo/
[2] https://www.appveyor.com/docs/packaging-artifacts
[3] https://www.youtube.com/watch?v=d-p6lJJObLU

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Nick Coghlan]

On 22 January 2016 at 20:48, M.-A. Lemburg  wrote:
> People who rely on Linux distributions want to continue
> to do so and get regular updates for system packages from
> their system vendor. Having wheel files override these
> system packages by including libs directly in the wheel
> silently breaks this expectation, potentially opening
> up the installations for security holes, difficult to
> track bugs and possible version conflicts with already
> loaded versions of the shared libs.

For the time being, these users should either pass the "--no-binary"
option to pip, ask their distro to provide an index of pre-built wheel
files for that distro (once we have the distro-specific wheel tagging
PEP sorted out), or else ask their distro to update system Python
packages in a more timely fashion (or all of the above).

> IMO, that's much worse than having to install additional
> system packages to make a Python wheel work.
>
> The embedding approach also creates licensing problems,
> since those libs may be under different licenses than the
> package itself. And of course, it increases the size of the
> wheel files, causing more bandwidth to be necessary,
> more disk space to be used for wheel caches, etc.

Then *don't publish manylinux wheel files*. Manylinux is, by design, a
platform+publisher-silo model, very similar to the way smart phone
operating systems work, and the way Windows applications and (I
believe) Mac App Bundles work. It is anti-thetical to the traditional
tightly coupled shared everything model adopted by Linux distributions
(where all the binaries are also generally built by a common central
build system).

There is a different model, which could be tagged as (for example)
"integratedlinux1", which is the one you propose. That wouldn't be
viable from a UX perspective without an external dependency
description system like the one Tennessee proposed in
https://github.com/pypa/interoperability-peps/pull/30, but that
approach requires a lot more development work before it could be
adopted.

>From the point of view of future-proofing PEP 513 against having such
an alternative available in the future, the main question that would
need to be considered is how tools would decide download priority
between a distro-specific wheel, an integrated linux wheel, and a
linux wheel with bundled dependencies.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Nick Coghlan]

On 22 January 2016 at 21:25, Donald Stufft  wrote:
> Thinking of it in terms of a C like "undefined behavior" is probably a
> reasonable way of doing it. Linking against a system provided library that is
> on this list is a defined behavior of the manylinux "platform", linking 
> against
> something else is undefined and may or may not work. At some level, once 
> you've
> gotten to the point you're using pip to manage some set of your packages it
> doesn't really matter if that set of things you're pulling from PyPI includes
> a C library or not. If you're relying on say psycopg2 it's not clear to me 
> that
> libpq *needs* to be getting security any more than psycopg2 itself does and so
> you'll need some method of solving that problem for your Python level
> dependencies anyways.

It also wouldn't surprise me if CVE trackers like requires.io and
versioneye.com gained the ability to search wheel files for embedded
dependencies and flag outdated and vulnerable ones.

However, it's a good point that PyPI won't be running auditwheel to
*force* compliance with the "no external dependencies outside the
defined set" guideline - while a service like pythonwheels.com could
potentially be set up independently of PyPI to run auditwheel on
manylinux wheels, PyPI itself wouldn't do it.

An external scan like that could actually be a useful way of defining
manylinux2 in the future - scanning popular manylinux wheel downloads
for both embedded libraries and for external dependencies outside the
defined set.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Donald Stufft]

> On Jan 22, 2016, at 6:29 AM, Nick Coghlan  wrote:
> 
> On 22 January 2016 at 20:48, M.-A. Lemburg  wrote:
>> People who rely on Linux distributions want to continue
>> to do so and get regular updates for system packages from
>> their system vendor. Having wheel files override these
>> system packages by including libs directly in the wheel
>> silently breaks this expectation, potentially opening
>> up the installations for security holes, difficult to
>> track bugs and possible version conflicts with already
>> loaded versions of the shared libs.
> 
> For the time being, these users should either pass the "--no-binary"
> option to pip, ask their distro to provide an index of pre-built wheel
> files for that distro (once we have the distro-specific wheel tagging
> PEP sorted out), or else ask their distro to update system Python
> packages in a more timely fashion (or all of the above).

I should note, once we have some solution to the fact that "linux, 64bit" is
way too generic of a platform tag for the general case I plan to allow the
current super generic platform tags to be uploaded as well to PyPI. We don't
try to prevent you from ever being able to release a broken package, we just
want to make it reasonable that you can do the right thing. In other words, as
long as the tooling makes it possible to do the right thing, the fact that you
can also generate packaging bugs in your project (as opposed to pip doing the
wrong thing) isn't a problem. So if people want to do something that isn't
manylinux and don't want to claim to be a manylinux wheel, they'll be free to
use the current linux tags as well.

> 
>> IMO, that's much worse than having to install additional
>> system packages to make a Python wheel work.
>> 
>> The embedding approach also creates licensing problems,
>> since those libs may be under different licenses than the
>> package itself. And of course, it increases the size of the
>> wheel files, causing more bandwidth to be necessary,
>> more disk space to be used for wheel caches, etc.
> 
> Then *don't publish manylinux wheel files*. Manylinux is, by design, a
> platform+publisher-silo model, very similar to the way smart phone
> operating systems work, and the way Windows applications and (I
> believe) Mac App Bundles work. It is anti-thetical to the traditional
> tightly coupled shared everything model adopted by Linux distributions
> (where all the binaries are also generally built by a common central
> build system).
> 
> There is a different model, which could be tagged as (for example)
> "integratedlinux1", which is the one you propose. That wouldn't be
> viable from a UX perspective without an external dependency
> description system like the one Tennessee proposed in
> https://github.com/pypa/interoperability-peps/pull/30, but that
> approach requires a lot more development work before it could be
> adopted.
> 
> From the point of view of future-proofing PEP 513 against having such
> an alternative available in the future, the main question that would
> need to be considered is how tools would decide download priority
> between a distro-specific wheel, an integrated linux wheel, and a
> linux wheel with bundled dependencies.

I think that this could probably just be left up to the individual tools? We
already have to decide between multiple candidate wheels, this is just another
thing to look at.

> 
> Regards,
> Nick.
> 
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Paul Moore]

On 22 January 2016 at 11:55, Donald Stufft  wrote:
>> From the point of view of future-proofing PEP 513 against having such
>> an alternative available in the future, the main question that would
>> need to be considered is how tools would decide download priority
>> between a distro-specific wheel, an integrated linux wheel, and a
>> linux wheel with bundled dependencies.
>
> I think that this could probably just be left up to the individual tools? We
> already have to decide between multiple candidate wheels, this is just another
> thing to look at.

The compatibility tag spec (PEP 425) does note that choosing between
candidate implementations is a tool-specific implementation detail (at
https://www.python.org/dev/peps/pep-0425/#id14). It seems to me that
this simply comes under the point of "It is recommended that
installers try to choose the most feature complete built distribution
available (the one most specific to the installation environment)..."

At the moment, the process is effective but rudimentary (we don't
really expect more than one compatible wheel that isn't pure-python).
If people start seeing multiple potential wheels and want to have the
option to control the priorities, it's probably better to develop a
solution via pip feature requests, and when things stabilise, document
the solution in a standard if it's appropriate (i.e., it's *not* just
pip configuration options).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Donald Stufft]

> On Jan 22, 2016, at 5:48 AM, M.-A. Lemburg  wrote:
> 
> Embedding additional libraries in the wheels files to overcome
> deficiencies in the PEP design simply doesn't feel right
> to me.
> 
> People who rely on Linux distributions want to continue
> to do so and get regular updates for system packages from
> their system vendor. Having wheel files override these
> system packages by including libs directly in the wheel
> silently breaks this expectation, potentially opening
> up the installations for security holes, difficult to
> track bugs and possible version conflicts with already
> loaded versions of the shared libs.
> 
> IMO, that's much worse than having to install additional
> system packages to make a Python wheel work.
> 
> The embedding approach also creates licensing problems,
> since those libs may be under different licenses than the
> package itself. And of course, it increases the size of the
> wheel files, causing more bandwidth to be necessary,
> more disk space to be used for wheel caches, etc.

I think there are a few things here, but this is not my area of expertise so I
could be wrong. As I understand it, The manylinux platform definition is
largely going to be a documentation effort and there isn't going to be much in
the way of enforcement. That means that people who build wheels against the
manylinux platform tag are free to really do whatever they want even if it
doesn't strictly match the definition of the manylinux platform. The difference
here is that if you link against something that isn't included in the set of
libraries, and that subsequently breaks due to an ABI incompatability, that's
not a pip bug or a manylinux bug, that's a packaging bug with that particular
library and they'll have to decide how they want to resolve it (if they want
to resolve it). So you'll be free to link to anything you want, but you get to
keep both pieces if it breaks and it's outside this defined set of libraries.

I also agree that it's OK for users to have to ``apt-get`` (or whatever) a
particular library to use something and we don't have to *only* rely on items
that are installed as part of a "standard" linux base system. However, what is
not OK (IMO) is for the PEP to bless something that has a high chance of ending
up with ABI issues rather than "need to apt-get install" issues. For instance,
even if you compile against a sufficiently old copy of OpenSSL, OpenSSL (to my
understanding) does not have a stable ABI and you cannot take something
compiled against OpenSSL on CentOS 5.reallyold and expect it to work on say
Arch Linux.

So I think there's an explicit list of packages that we know will generally
work as long as you build against a sufficiently old copy of them and outside
of that it's really a big unknown in general if a particular library can be
used in this way or not. We obviously can't enumerate the list of every
possible C library that has a stable ABI that can sanely be used cross distro
but I think it's reasonable to list some sort of base minimum here, and if
people experiment with stepping outside the standard list and can come to us
and show "hey, I tried it with xyz library, we've gotten X installs and no
complaints" we can then possibly expand the definition of the manylinux
platform to include that library and move that project from depending on
undefined behavior to defined behavior.

Thinking of it in terms of a C like "undefined behavior" is probably a
reasonable way of doing it. Linking against a system provided library that is
on this list is a defined behavior of the manylinux "platform", linking against
something else is undefined and may or may not work. At some level, once you've
gotten to the point you're using pip to manage some set of your packages it
doesn't really matter if that set of things you're pulling from PyPI includes
a C library or not. If you're relying on say psycopg2 it's not clear to me that
libpq *needs* to be getting security any more than psycopg2 itself does and so
you'll need some method of solving that problem for your Python level
dependencies anyways.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[M.-A. Lemburg]

On 22.01.2016 12:25, Donald Stufft wrote:
> 
>> On Jan 22, 2016, at 5:48 AM, M.-A. Lemburg  wrote:
>>
>> Embedding additional libraries in the wheels files to overcome
>> deficiencies in the PEP design simply doesn't feel right
>> to me.
>>
>> People who rely on Linux distributions want to continue
>> to do so and get regular updates for system packages from
>> their system vendor. Having wheel files override these
>> system packages by including libs directly in the wheel
>> silently breaks this expectation, potentially opening
>> up the installations for security holes, difficult to
>> track bugs and possible version conflicts with already
>> loaded versions of the shared libs.
>>
>> IMO, that's much worse than having to install additional
>> system packages to make a Python wheel work.
>>
>> The embedding approach also creates licensing problems,
>> since those libs may be under different licenses than the
>> package itself. And of course, it increases the size of the
>> wheel files, causing more bandwidth to be necessary,
>> more disk space to be used for wheel caches, etc.
> 
> I think there are a few things here, but this is not my area of expertise so I
> could be wrong. As I understand it, The manylinux platform definition is
> largely going to be a documentation effort and there isn't going to be much in
> the way of enforcement. That means that people who build wheels against the
> manylinux platform tag are free to really do whatever they want even if it
> doesn't strictly match the definition of the manylinux platform. The 
> difference
> here is that if you link against something that isn't included in the set of
> libraries, and that subsequently breaks due to an ABI incompatability, that's
> not a pip bug or a manylinux bug, that's a packaging bug with that particular
> library and they'll have to decide how they want to resolve it (if they want
> to resolve it). So you'll be free to link to anything you want, but you get to
> keep both pieces if it breaks and it's outside this defined set of libraries.

Hmm, if that were the reading, things would look a lot brighter,
but if PyPI will start to only support uploading manylinux wheels
for Linux platforms, you essentially have the effect that the
PEP ends up defining the set of allowed external libraries and forces
package authors to embed any other external libraries into the
wheel file - or not be able to upload wheel files for Linux at all.

This can hardly be in the interest of Python users who don't want
to use wheel embedded system libraries on their Linux system and
most likely also don't expect wheel files to ship alternative
versions with them in the first place.

If we'd lift the ban of "linux_*" tagged wheels on PyPI at
the same time we allow "manylinux" wheels, that'd remove a lot
of my concerns.

In that case, I'd just like to see a way to tell pip not to install
manylinux wheels with embedded system libraries, or simply outright
reject embedded system libraries in manylinux wheel files.

> I also agree that it's OK for users to have to ``apt-get`` (or whatever) a
> particular library to use something and we don't have to *only* rely on items
> that are installed as part of a "standard" linux base system. However, what is
> not OK (IMO) is for the PEP to bless something that has a high chance of 
> ending
> up with ABI issues rather than "need to apt-get install" issues. For instance,
> even if you compile against a sufficiently old copy of OpenSSL, OpenSSL (to my
> understanding) does not have a stable ABI and you cannot take something
> compiled against OpenSSL on CentOS 5.reallyold and expect it to work on say
> Arch Linux.

True. There will always be incompatibilities out there which
cannot be addressed with a one-fits-all approach. For those
cases, vendor specific wheels would need to be created.

> So I think there's an explicit list of packages that we know will generally
> work as long as you build against a sufficiently old copy of them and outside
> of that it's really a big unknown in general if a particular library can be
> used in this way or not. We obviously can't enumerate the list of every
> possible C library that has a stable ABI that can sanely be used cross distro
> but I think it's reasonable to list some sort of base minimum here, and if
> people experiment with stepping outside the standard list and can come to us
> and show "hey, I tried it with xyz library, we've gotten X installs and no
> complaints" we can then possibly expand the definition of the manylinux
> platform to include that library and move that project from depending on
> undefined behavior to defined behavior.
> 
> Thinking of it in terms of a C like "undefined behavior" is probably a
> reasonable way of doing it. Linking against a system provided library that is
> on this list is a defined behavior of the manylinux "platform", linking 
> against
> something else is undefined and may or may not work. At some level, once 
> 

Re: [Distutils] draft PEP: manylinux1

[Lowekamp, Bradley (NIH/NLM/LHC) [C]]

Hello,

I noticed that libpython is missing from the lists of dependent libraries. Also 
the “manylinux” Docker image has it’s Python versions compiled with libpython 
static.

Does this mean that we must do static linking against libpython?
If so, can’t this cause problems with mixing Python objects from different 
patch versions at runtime? 

I checked Anaconda, and they are using a shared Python library.

Thanks,
Brad

From: Nathaniel Smith [n...@pobox.com]
Sent: Wednesday, January 20, 2016 10:55 PM
To: distutils-sig
Subject: [Distutils] draft PEP: manylinux1

Hi all,

Here's a first draft of a PEP for the manylinux1 platform tag
mentioned earlier, posted for feedback. Really Robert McGibbon should
get the main credit for this, since he wrote it, and also the docker
image and the amazing auditwheel tool linked below, but he asked me to
do the honors of posting it :-).

BTW, if anyone wants to try this out, there are some test
"manylinux1-compatible" wheels at
  https://vorpus.org/~njs/tmp/manylinux-test-wheels/repaired
for PySide (i.e. Qt) and numpy (using openblas). They should be
installable on any ordinary linux system with:
  pip install --no-index -f
https://vorpus.org/~njs/tmp/manylinux-test-wheels/repaired $PKG
(Note that this may require a reasonably up-to-date pip -- e.g. the
one in Debian is too old, which confused me for a bit.)

(How they were created: docker run -it quay.io/manylinux/manylinux
bash; install conda because to get builds of Qt and OpenBLAS because I
was too lazy to do it myself; pip wheel PySide / pip wheel numpy;
auditwheel repair , which copies in all the
dependencies to make the wheels self-contained. Just proof-of-concept
for now, but they seem to work.)



PEP: 
Title: A Platform Tag for Portable Linux Built Distributions
Version: $Revision$
Last-Modified: $Date$
Author: Robert T. McGibbon , Nathaniel J. Smith

Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 19-Jan-2016
Post-History: 19-Jan-2016


Abstract


This PEP proposes the creation of a new platform tag for Python package built
distributions, such as wheels, called ``manylinux1_{x86_64,i386}`` with
external dependencies limited restricted to a standardized subset of
the Linux kernel and core userspace ABI. It proposes that PyPI support
uploading and distributing Wheels with this platform tag, and that ``pip``
support downloading and installing these packages on compatible platforms.


Rationale
=

Currently, distribution of binary Python extensions for Windows and OS X is
straightforward. Developers and packagers build wheels, which are assigned
platform tags such as ``win32`` or ``macosx_10_6_intel``, and upload these
wheels to PyPI. Users can download and install these wheels using tools such
as ``pip``.

For Linux, the situation is much more delicate. In general, compiled Python
extension modules built on one Linux distribution will not work on other Linux
distributions, or even on the same Linux distribution with different system
libraries installed.

Build tools using PEP 425 platform tags [1]_ do not track information about the
particular Linux distribution or installed system libraries, and instead assign
all wheels the too-vague ``linux_i386`` or ``linux_x86_64`` tags. Because of
this ambiguity, there is no expectation that ``linux``-tagged built
distributions compiled on one machine will work properly on another, and for
this reason, PyPI has not permitted the uploading of wheels for Linux.

It would be ideal if wheel packages could be compiled that would work on *any*
linux system. But, because of the incredible diversity of Linux systems -- from
PCs to Android to embedded systems with custom libcs -- this cannot
be guaranteed in general.

Instead, we define a standard subset of the kernel+core userspace ABI that,
in practice, is compatible enough that packages conforming to this standard
will work on *many* linux systems, including essentially all of the desktop
and server distributions in common use. We know this because there are
companies who have been distributing such widely-portable pre-compiled Python
extension modules for Linux -- e.g. Enthought with Canopy [2]_ and Continuum
Analytics with Anaconda [3]_.

Building on the compability lessons learned from these companies, we thus
define a baseline ``manylinux1`` platform tag for use by binary Python
wheels, and introduce the implementation of preliminary tools to aid in the
construction of these ``manylinux1`` wheels.


Key Causes of Inter-Linux Binary Incompatibility


To properly define a standard that will guarantee that wheel packages meeting
this specification will operate on *many* linux platforms, it is necessary to
understand the root causes which often prevent portability of pre-compiled
binaries on Linux. The two key causes are dependencies on shared libraries
which are not 

Re: [Distutils] Package classifiers - both major and minor Python versions?

[Fred Drake]

On Fri, Jan 22, 2016 at 2:57 AM, Chris Jerdonek
 wrote:
> If "Python 2" really means that all 2.x versions are supported as
> opposed to just some, then that should also be clarified (and
> similarly for "Python 3").

Bingo!  That's the main point, I think.  Because there's no single
reference for what it means to apply any particular classifier, we're
not going to get a lot of mileage from them.

At this point, classifiers are still pretty much recreational.


  -Fred

-- 
Fred L. Drake, Jr.
"A storm broke loose in my mind."  --Albert Einstein
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Nick Coghlan]

On 23 January 2016 at 08:17, Nathaniel Smith  wrote:
> On Fri, Jan 22, 2016 at 11:28 AM, Donald Stufft  wrote:
>> Yea, that’s why I thought about dpkg (system) or system(Debian) or
>> something. The main reason I can think of for preferring “system” is if we
>> don’t want to change the valid characters for a value in this file. Then you
>> can have system(Debian) and system(Conda) and everything will work just
>> fine.
>
> Maybe to simplify the discussion we should just forget about INSTALLER,
> leave it in peace to be what the PEP says ("fyi this is the last program to
> touch this, in case anyone cares"), and add a new file with whatever
> syntax/semantics make sense. Filenames are cheap and plentiful, and no
> reason to complicate this discussion with hypothetical backcompat worries.

Right, that was my theory in implementing INSTALLER just as PEP 376
defined it - having pip write "pip" to that file is enough to let us
determine:

1. Controlled in a pip-compatible way
2. Controlled in a pip-incompatible way
3. pip compatibility not specified

Paul's right that at the time the assumption was each tool would just
write *its own* name into the file, but given the other changes that
have happened since then, it now makes sense that other tools that are
fully interoperable with another more-widely used installer may choose
to write that installer's name instead of their own, and the only
states we *really* care about from a UX perspective are "definitely
compatible", "definitely incompatible" and "don't know".

We may decide to do something more sophisticated later, but that would
be as part of someone finding the roundtuits to actually design a
plugin system for delegating to system package managers when running
as root.

From a downstream perspective, the main thing we need to know in order
to choose a suitable value to put into that file is how pip decides to
use whatever we write there. For example, if pip were to emit a
warning message looking something like:

" is not managed by pip, skipping (use ''
or '--force' to update it)"

Then the distro could update our packaging policies such that we write
our respective package management command names into the INSTALLER
files when creating system packages.

(As far as the regex defining the permitted content goes,
appropriately caveating PEP 376 to better match the reality of how
current generation tools work is one of my main motivations for
revising the specification process)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 376, the INSTALLER file, and system packages

[Jeremy Stanley]

On 2016-01-23 08:09:47 +1300 (+1300), Robert Collins wrote:
[...]
> Having a interface to the system package manager as has been
> previously mooted - and I'm going to get back to that - might help a
> little, but uninstalls are quite different to installs. Uninstalls get
> blocked by things like 'app-foo requires requests', and I would be
> super suprised (and delighted!) if we were able to override that when
> upgrading requests via pip :)

Add to this the fact that whatever system-distributed files pip
removes/replaces will get quietly overwritten again the next time
you update your system packages and there's a new version of
whatever you've replaced... perhaps being able to set a hold state
through the distro package manager from pip at that point for
whatever package owned the files it replaced would be a helpful
middle ground (I know how I'd go about it on Debian derivatives,
less so on all those "other" distros).
-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig