Re: [Distutils] pip/warehouse feature idea: "help needed"

[Olivier Grisel]

+1 overall to Nick' suggestions.

-- 
Olivier
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

Hi there,

During pycon, Nick mentioned there was interest in updating the wheel
format to support downstream distributions. Nick mentioned Linux
distributions, but I would like to express interest for other kind of
downstream distributors like Anaconda from Continuum or Canopy from
Enthought (disclaimer: I work for Enthought).

Right now, wheels have the following limitations for us:

1. lack of post/pre install/removing
2. more fine-grained installation scheme
3. lack of clarify on which tags vendors should use for custom wheels: some
packages we provide would not be installable on "normal" python, and it
would be nice to have a scheme to avoid confusion there as well.

At least 1. and 2. are of interest not just for us.

Regarding 2., it looks like anything in the .data/data
directory will be placed as is in sys.prefix by pip. This is how distutils
scheme is defined ATM, but I am not sure whether that's by design or
accident ?

I would suggest to use something close to autotools, with some tweaks to
work well on windows. I implemented something like this in my project bento
(
https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py),
but we could of course tweak that.

For 1., I believe it was a conscious decision not to include them in wheel
1.0 ? Would it make sense to start a discussion to add it to wheel ?

I will be at the pycon sprints until wednesday evening, so that we can
flesh some concrete proposal first, if there is enough interest.

As a background: at Enthought, we have been using eggs to distribute
binaries of python packages and other packages (e.g. C libraries, compiled
binaries, etc...) for a very long time. We had our own extensions to the
egg format to support this, but I want to get out of eggs so as to make our
own software more compatible with where the community is going. I would
also like to avoid making ad-hoc extensions to wheels for our own purposes.

thanks,
David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Donald Stufft]

> On Apr 13, 2015, at 10:39 AM, David Cournapeau  wrote:
> 
> Hi there,
> 
> During pycon, Nick mentioned there was interest in updating the wheel format 
> to support downstream distributions. Nick mentioned Linux distributions, but 
> I would like to express interest for other kind of downstream distributors 
> like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work for 
> Enthought).
> 
> Right now, wheels have the following limitations for us:
> 
> 1. lack of post/pre install/removing
> 2. more fine-grained installation scheme
> 3. lack of clarify on which tags vendors should use for custom wheels: some 
> packages we provide would not be installable on "normal" python, and it would 
> be nice to have a scheme to avoid confusion there as well.
> 
> At least 1. and 2. are of interest not just for us.
> 
> Regarding 2., it looks like anything in the .data/data directory 
> will be placed as is in sys.prefix by pip. This is how distutils scheme is 
> defined ATM, but I am not sure whether that's by design or accident ?
> 
> I would suggest to use something close to autotools, with some tweaks to work 
> well on windows. I implemented something like this in my project bento 
> (https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py
>  
> ),
>  but we could of course tweak that.
> 
> For 1., I believe it was a conscious decision not to include them in wheel 
> 1.0 ? Would it make sense to start a discussion to add it to wheel ?
> 
> I will be at the pycon sprints until wednesday evening, so that we can flesh 
> some concrete proposal first, if there is enough interest.
> 
> As a background: at Enthought, we have been using eggs to distribute binaries 
> of python packages and other packages (e.g. C libraries, compiled binaries, 
> etc...) for a very long time. We had our own extensions to the egg format to 
> support this, but I want to get out of eggs so as to make our own software 
> more compatible with where the community is going. I would also like to avoid 
> making ad-hoc extensions to wheels for our own purposes.
> 

To my knowledge, (1) was purposely punted until a later revision of Wheel just 
to make it easier to land the “basic” wheel.

I think (2) is a reasonable thing as long as we can map it sanely on all 
platforms.

I’m not sure what (3) means exactly. What is a “normal” Python, do you modify 
Python in a way that breaks the ABI but which isn’t reflected in the standard 
ABI tag?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft  wrote:

>
> On Apr 13, 2015, at 10:39 AM, David Cournapeau  wrote:
>
> Hi there,
>
> During pycon, Nick mentioned there was interest in updating the wheel
> format to support downstream distributions. Nick mentioned Linux
> distributions, but I would like to express interest for other kind of
> downstream distributors like Anaconda from Continuum or Canopy from
> Enthought (disclaimer: I work for Enthought).
>
> Right now, wheels have the following limitations for us:
>
> 1. lack of post/pre install/removing
> 2. more fine-grained installation scheme
> 3. lack of clarify on which tags vendors should use for custom wheels:
> some packages we provide would not be installable on "normal" python, and
> it would be nice to have a scheme to avoid confusion there as well.
>
> At least 1. and 2. are of interest not just for us.
>
> Regarding 2., it looks like anything in the .data/data
> directory will be placed as is in sys.prefix by pip. This is how distutils
> scheme is defined ATM, but I am not sure whether that's by design or
> accident ?
>
> I would suggest to use something close to autotools, with some tweaks to
> work well on windows. I implemented something like this in my project bento
> (
> https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py),
> but we could of course tweak that.
>
> For 1., I believe it was a conscious decision not to include them in wheel
> 1.0 ? Would it make sense to start a discussion to add it to wheel ?
>
> I will be at the pycon sprints until wednesday evening, so that we can
> flesh some concrete proposal first, if there is enough interest.
>
> As a background: at Enthought, we have been using eggs to distribute
> binaries of python packages and other packages (e.g. C libraries, compiled
> binaries, etc...) for a very long time. We had our own extensions to the
> egg format to support this, but I want to get out of eggs so as to make our
> own software more compatible with where the community is going. I would
> also like to avoid making ad-hoc extensions to wheels for our own purposes.
>
>
> To my knowledge, (1) was purposely punted until a later revision of Wheel
> just to make it easier to land the “basic” wheel.
>

Great. Was there any proposal made to support it at all ? Or should I just
work from scratch there ?


>
> I think (2) is a reasonable thing as long as we can map it sanely on all
> platforms.
>

Yes. We support all platforms at Enthought, and Windows is important for us
!


> I’m not sure what (3) means exactly. What is a “normal” Python, do you
> modify Python in a way that breaks the ABI but which isn’t reflected in the
> standard ABI tag?
>

It could be multiple things. The most obvious one is that generally.
cross-platforms python distributions will try to be "relocatable" (i.e. the
whole installation can be moved and still work). This means they require
python itself to be built a special way. Strictly speaking, it is not an
ABI issue, but the result is the same though: you can't use libraries from
anaconda or canopy on top of a normal python

More generally, we could be modifying python in a way that is not forward
compatible with upstream python: a binary that works on our python may not
work on the python from python.org (though the opposite is true). It would
be nice if one could make sure pip will not try to install those eggs when
installed on top of a python that does not advertise itself as "compatible".

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Daniel Holth]

#1 is pretty straightforward. An entry-point format Python
pre/post/etc. script may do.

I have some ideas for the FHS, though I fear it's full of bikesheds:

1. Allow all GNU directory variables as .data/* subdirectories
(https://www.gnu.org/prep/standards/html_node/Directory-Variables.html).
The distutils names will continue to be allowed.

packagename-1.0.data/mandir/...

2. Make data_files useful again. Interpolate path variables into
distutils data_files using $template syntax. (Only allow at
beginning?)

data_files=[('$mandir/xyz', ['manfile', 'other_man_file'])

In addition to $bindir, $mandir, etc. it will be important to allow
the package name and version to be interpolated into the install
directories.

Inside the wheel archive, you will get
packagename-1.0.data/mandir/manfile and
packagename-1.0.data/mandir/other_man_file

3. Write the install paths (the mapping from $bindir, $mandir, $prefix
etc. to the actual paths used) to one or more of a .py, .json, or
.dist-info/* based on new metadata in WHEEL:

install-paths-to: wheel/_paths.py

It is critical that this be allowed to work without requiring the end
user to look for it with pkg_resources or its pals. It's also good to
only write it if the installed package actually needs to locate its
file categories after it has been installed.

This will also be written inside the wheel itself with relative paths
to the .data/ directory.

4. Allow configurable & custom paths. The GNU paths could be
configured relative to the distutils paths as a default. We might let
the user add additional paths with a configuration dict.

paths = {
 "foo" : "$bar/${quux}",
 "bar" : "${baz}/more/stuff",
 "baz" : "${quux}/again",
 "quux": "larry"
 }

5. On Windows, no one will really care where most of these files go,
but they probably won't mind if they are installed into separate
directories. Come up with sensible locations for the most important
categories.


On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft  wrote:
>
> On Apr 13, 2015, at 10:39 AM, David Cournapeau  wrote:
>
> Hi there,
>
> During pycon, Nick mentioned there was interest in updating the wheel format
> to support downstream distributions. Nick mentioned Linux distributions, but
> I would like to express interest for other kind of downstream distributors
> like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work
> for Enthought).
>
> Right now, wheels have the following limitations for us:
>
> 1. lack of post/pre install/removing
> 2. more fine-grained installation scheme
> 3. lack of clarify on which tags vendors should use for custom wheels: some
> packages we provide would not be installable on "normal" python, and it
> would be nice to have a scheme to avoid confusion there as well.
>
> At least 1. and 2. are of interest not just for us.
>
> Regarding 2., it looks like anything in the .data/data directory
> will be placed as is in sys.prefix by pip. This is how distutils scheme is
> defined ATM, but I am not sure whether that's by design or accident ?
>
> I would suggest to use something close to autotools, with some tweaks to
> work well on windows. I implemented something like this in my project bento
> (https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py),
> but we could of course tweak that.
>
> For 1., I believe it was a conscious decision not to include them in wheel
> 1.0 ? Would it make sense to start a discussion to add it to wheel ?
>
> I will be at the pycon sprints until wednesday evening, so that we can flesh
> some concrete proposal first, if there is enough interest.
>
> As a background: at Enthought, we have been using eggs to distribute
> binaries of python packages and other packages (e.g. C libraries, compiled
> binaries, etc...) for a very long time. We had our own extensions to the egg
> format to support this, but I want to get out of eggs so as to make our own
> software more compatible with where the community is going. I would also
> like to avoid making ad-hoc extensions to wheels for our own purposes.
>
>
> To my knowledge, (1) was purposely punted until a later revision of Wheel
> just to make it easier to land the “basic” wheel.
>
> I think (2) is a reasonable thing as long as we can map it sanely on all
> platforms.
>
> I’m not sure what (3) means exactly. What is a “normal” Python, do you
> modify Python in a way that breaks the ABI but which isn’t reflected in the
> standard ABI tag?
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Daniel Holth]

On Mon, Apr 13, 2015 at 10:54 AM, David Cournapeau  wrote:
>
>
> On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft  wrote:
>>
>>
>> On Apr 13, 2015, at 10:39 AM, David Cournapeau  wrote:
>>
>> Hi there,
>>
>> During pycon, Nick mentioned there was interest in updating the wheel
>> format to support downstream distributions. Nick mentioned Linux
>> distributions, but I would like to express interest for other kind of
>> downstream distributors like Anaconda from Continuum or Canopy from
>> Enthought (disclaimer: I work for Enthought).
>>
>> Right now, wheels have the following limitations for us:
>>
>> 1. lack of post/pre install/removing
>> 2. more fine-grained installation scheme
>> 3. lack of clarify on which tags vendors should use for custom wheels:
>> some packages we provide would not be installable on "normal" python, and it
>> would be nice to have a scheme to avoid confusion there as well.
>>
>> At least 1. and 2. are of interest not just for us.
>>
>> Regarding 2., it looks like anything in the .data/data
>> directory will be placed as is in sys.prefix by pip. This is how distutils
>> scheme is defined ATM, but I am not sure whether that's by design or
>> accident ?
>>
>> I would suggest to use something close to autotools, with some tweaks to
>> work well on windows. I implemented something like this in my project bento
>> (https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py),
>> but we could of course tweak that.
>>
>> For 1., I believe it was a conscious decision not to include them in wheel
>> 1.0 ? Would it make sense to start a discussion to add it to wheel ?
>>
>> I will be at the pycon sprints until wednesday evening, so that we can
>> flesh some concrete proposal first, if there is enough interest.
>>
>> As a background: at Enthought, we have been using eggs to distribute
>> binaries of python packages and other packages (e.g. C libraries, compiled
>> binaries, etc...) for a very long time. We had our own extensions to the egg
>> format to support this, but I want to get out of eggs so as to make our own
>> software more compatible with where the community is going. I would also
>> like to avoid making ad-hoc extensions to wheels for our own purposes.
>>
>>
>> To my knowledge, (1) was purposely punted until a later revision of Wheel
>> just to make it easier to land the “basic” wheel.
>
>
> Great. Was there any proposal made to support it at all ? Or should I just
> work from scratch there ?
>
>>
>>
>> I think (2) is a reasonable thing as long as we can map it sanely on all
>> platforms.
>
>
> Yes. We support all platforms at Enthought, and Windows is important for us
> !
>
>>
>> I’m not sure what (3) means exactly. What is a “normal” Python, do you
>> modify Python in a way that breaks the ABI but which isn’t reflected in the
>> standard ABI tag?
>
>
> It could be multiple things. The most obvious one is that generally.
> cross-platforms python distributions will try to be "relocatable" (i.e. the
> whole installation can be moved and still work). This means they require
> python itself to be built a special way. Strictly speaking, it is not an ABI
> issue, but the result is the same though: you can't use libraries from
> anaconda or canopy on top of a normal python
>
> More generally, we could be modifying python in a way that is not forward
> compatible with upstream python: a binary that works on our python may not
> work on the python from python.org (though the opposite is true). It would
> be nice if one could make sure pip will not try to install those eggs when
> installed on top of a python that does not advertise itself as "compatible"

We need a hook to alter pip's list of compatible tags
(pip.pep425tags), and to alter the default tags used by bdist_wheel
when creating wheels. One sensible proposal for "special" wheels is to
just use a truncated hash of the platform description (a random hex
string) in place of the wheel platform tag.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Paul Moore]

On 13 April 2015 at 16:02, Daniel Holth  wrote:
> #1 is pretty straightforward. An entry-point format Python
> pre/post/etc. script may do.

There's metadata 2.0 information for this. It would be sensible to
follow that definition where it applies, but otherwise yes, this
shouldn't be hard.

Some thoughts, though:

1. Some thought should be put into how we ensure that pre/post
install/remove scripts are cross-platform. It would be a shame if a
wheel was unusable on Windows for no reason other than that the
postinstall script was written as a bash script. Or on Unix because
the postinstall script tried to write Windows start menu items.
2. It's worth considering "appropriate use" of such scripts. The
Windows start menu example is relevant here - I can easily imagine
users requesting something like that for a wheel they want to install
into the system Python, but it's completely inappropriate for
installing into a virtualenv.

To an extent, there's nothing we can (or maybe even should) do about
this - projects that include inappropriate install scripts will get
issues raised or will lose users, and the problem is self-correcting
to an extent, but it's probably worth including in the implementation,
some work to add appropriate documentation to the packaging user guide
about "best practices" for pre/post-install/remove scripts (hmm, a
glossary entry with a good name for these beasts would also be helpful
:-))

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Chris Barker]

NOTE: I don't work for any of the companies involved -- just a somewhat
frustrated user... And someone that has been trying for years to make
things easier for OS-X users.

I’m not sure what (3) means exactly. What is a “normal” Python, do you
>> modify Python in a way that breaks the ABI but which isn’t reflected in the
>> standard ABI tag?
>>
>
> It could be multiple things. The most obvious one is that generally.
> cross-platforms python distributions will try to be "relocatable" (i.e. the
> whole installation can be moved and still work). This means they require
> python itself to be built a special way. Strictly speaking, it is not an
> ABI issue, but the result is the same though: you can't use libraries from
> anaconda or canopy on top of a normal python
>

But why not? -- at least for Anaconda, it's because those libraries likely
have non-python dependencies, which are expected to be installed in a
particular way. And really, this is not particular to Anaconda/Canopy at
all. Python itself has no answer for this issue, and eggs and wheels don't
help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order
to build a binary wheel with non-python dependencies (let's say something
like libjpeg, for instance), you need to either:
 - assume that libjpeg is installed in a "standard" place -- really no
solution at all (at least outside of linux)
 - statically link it
 - ship the dynamic lib with the package

For the most part, the accepted solution for OS-X has been to statically
link, but:

 - it's a pain to do. The gnu toolchain really likes to use dynamic
linking, and building a static lib that will run on a
maybe-older-than-the-build-system machine is pretty tricky.

 - now we end up with multiple copies of the same lib in the python
install. There are a handful of libs that are used a LOT. Maybe there is no
real downside -- disk space and memory are cheap these days, but it sure
feels ugly. And I have yet to feel comfortable with having multiple
versions of the same lib linked into one python instance -- I can't say
I've seen a problem, but it makes me nervous.

On Windows, the choices are the same, except that: It is so much harder to
build many of the "standard" open source libs that package authors are more
likely to do it for folks, and you do get the occasional "dll hell" issues.

I had a plan to make some binary wheels for OS-X that were not really
python packages, but actually just bundled up libs, so that other wheels
could depend on them. OS-X does allow linking to relative paths, so this
should have been doable, but I never got anyone else to agree this was a
good idea, and I never found the roundtoits anyway. And it doesn't really
fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are
platform dependent and even worse, build-dependent.

Meanwhile, conda was chugging along and getting a lot of momentum in the
Scientific community. And the core thing here is that conda was designed
from the ground up to support essentially anything, This means is supports
python packages that depend on non-python packages, but also supports
packages that have nothing to do with python (Perl, command line tools,
what have you...)

So I have been focusing on conda lately.

Which brings me back to the question: should the python tools (i.e. wheel)
be extended to support more use-cases, specifically non-python
dependencies? Or do we just figure that that's a problem better solved by
projects with a larger scope (i.e. rpm, deb, conda, canopy).

I'm on the fence here. I mostly care about Python, and I think we're pretty
darn close with allowing wheel to support the non-python dependencies,
which would allow us all to "simply pip install" pretty much anything --
that would be cool. But maybe it's a bit of a slippery slope, and if we go
there, we'll end up re-writing conda.

BTW, while you can't generally install a conda package in/for another
python, you can generally install a wheel in a conda pythonThere are a
few issues with pip/setuptools trying to resolve dependencies while not
knowing about conda packages, but it does mostly work.

Not sure that helped the discussion -- but I've been wrestling with this
for a while, so thought I'd get my thoughts out there.


-Chris

















-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Daniel Holth]

On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker  wrote:
> NOTE: I don't work for any of the companies involved -- just a somewhat
> frustrated user... And someone that has been trying for years to make things
> easier for OS-X users.
>
>>> I’m not sure what (3) means exactly. What is a “normal” Python, do you
>>> modify Python in a way that breaks the ABI but which isn’t reflected in the
>>> standard ABI tag?
>>
>>
>> It could be multiple things. The most obvious one is that generally.
>> cross-platforms python distributions will try to be "relocatable" (i.e. the
>> whole installation can be moved and still work). This means they require
>> python itself to be built a special way. Strictly speaking, it is not an ABI
>> issue, but the result is the same though: you can't use libraries from
>> anaconda or canopy on top of a normal python
>
>
> But why not? -- at least for Anaconda, it's because those libraries likely
> have non-python dependencies, which are expected to be installed in a
> particular way. And really, this is not particular to Anaconda/Canopy at
> all. Python itself has no answer for this issue, and eggs and wheels don't
> help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to
> build a binary wheel with non-python dependencies (let's say something like
> libjpeg, for instance), you need to either:
>  - assume that libjpeg is installed in a "standard" place -- really no
> solution at all (at least outside of linux)
>  - statically link it
>  - ship the dynamic lib with the package
>
> For the most part, the accepted solution for OS-X has been to statically
> link, but:
>
>  - it's a pain to do. The gnu toolchain really likes to use dynamic linking,
> and building a static lib that will run on a
> maybe-older-than-the-build-system machine is pretty tricky.
>
>  - now we end up with multiple copies of the same lib in the python install.
> There are a handful of libs that are used a LOT. Maybe there is no real
> downside -- disk space and memory are cheap these days, but it sure feels
> ugly. And I have yet to feel comfortable with having multiple versions of
> the same lib linked into one python instance -- I can't say I've seen a
> problem, but it makes me nervous.
>
> On Windows, the choices are the same, except that: It is so much harder to
> build many of the "standard" open source libs that package authors are more
> likely to do it for folks, and you do get the occasional "dll hell" issues.
>
> I had a plan to make some binary wheels for OS-X that were not really python
> packages, but actually just bundled up libs, so that other wheels could
> depend on them. OS-X does allow linking to relative paths, so this should
> have been doable, but I never got anyone else to agree this was a good idea,
> and I never found the roundtoits anyway. And it doesn't really fit into the
> PyPi, pip, wheel, etc. philosphy to have dependencies that are platform
> dependent and even worse, build-dependent.
>
> Meanwhile, conda was chugging along and getting a lot of momentum in the
> Scientific community. And the core thing here is that conda was designed
> from the ground up to support essentially anything, This means is supports
> python packages that depend on non-python packages, but also supports
> packages that have nothing to do with python (Perl, command line tools, what
> have you...)
>
> So I have been focusing on conda lately.
>
> Which brings me back to the question: should the python tools (i.e. wheel)
> be extended to support more use-cases, specifically non-python dependencies?
> Or do we just figure that that's a problem better solved by projects with a
> larger scope (i.e. rpm, deb, conda, canopy).
>
> I'm on the fence here. I mostly care about Python, and I think we're pretty
> darn close with allowing wheel to support the non-python dependencies, which
> would allow us all to "simply pip install" pretty much anything -- that
> would be cool. But maybe it's a bit of a slippery slope, and if we go there,
> we'll end up re-writing conda.
>
> BTW, while you can't generally install a conda package in/for another
> python, you can generally install a wheel in a conda pythonThere are a
> few issues with pip/setuptools trying to resolve dependencies while not
> knowing about conda packages, but it does mostly work.
>
> Not sure that helped the discussion -- but I've been wrestling with this for
> a while, so thought I'd get my thoughts out there.

I've always thought of wheel as solving only the Python-specific
problem. Providing relocatable Python-specific packaging without
trying to solve the intractable problem of non-Python dependencies.
The strategy works best the more you are targeting "Python" as your
platform and not a specific OS or distribution - sometimes it works
well, other times not at all.

Obviously if you need a specific build of PostgreSQL wheel isn't going
to help you. With enough hacks you could make it work but are we ready
to "pip install kde"? I don

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker 
wrote:

> NOTE: I don't work for any of the companies involved -- just a somewhat
> frustrated user... And someone that has been trying for years to make
> things easier for OS-X users.
>
> I’m not sure what (3) means exactly. What is a “normal” Python, do you
>>> modify Python in a way that breaks the ABI but which isn’t reflected in the
>>> standard ABI tag?
>>>
>>
>> It could be multiple things. The most obvious one is that generally.
>> cross-platforms python distributions will try to be "relocatable" (i.e. the
>> whole installation can be moved and still work). This means they require
>> python itself to be built a special way. Strictly speaking, it is not an
>> ABI issue, but the result is the same though: you can't use libraries from
>> anaconda or canopy on top of a normal python
>>
>
> But why not? -- at least for Anaconda, it's because those libraries likely
> have non-python dependencies, which are expected to be installed in a
> particular way. And really, this is not particular to Anaconda/Canopy at
> all. Python itself has no answer for this issue, and eggs and wheels don't
> help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order
> to build a binary wheel with non-python dependencies (let's say something
> like libjpeg, for instance), you need to either:
>  - assume that libjpeg is installed in a "standard" place -- really no
> solution at all (at least outside of linux)
>  - statically link it
>  - ship the dynamic lib with the package
>
> For the most part, the accepted solution for OS-X has been to statically
> link, but:
>
>  - it's a pain to do. The gnu toolchain really likes to use dynamic
> linking, and building a static lib that will run on a
> maybe-older-than-the-build-system machine is pretty tricky.
>
>  - now we end up with multiple copies of the same lib in the python
> install. There are a handful of libs that are used a LOT. Maybe there is no
> real downside -- disk space and memory are cheap these days, but it sure
> feels ugly. And I have yet to feel comfortable with having multiple
> versions of the same lib linked into one python instance -- I can't say
> I've seen a problem, but it makes me nervous.
>
> On Windows, the choices are the same, except that: It is so much harder to
> build many of the "standard" open source libs that package authors are more
> likely to do it for folks, and you do get the occasional "dll hell" issues.
>
> I had a plan to make some binary wheels for OS-X that were not really
> python packages, but actually just bundled up libs, so that other wheels
> could depend on them. OS-X does allow linking to relative paths, so this
> should have been doable, but I never got anyone else to agree this was a
> good idea, and I never found the roundtoits anyway. And it doesn't really
> fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are
> platform dependent and even worse, build-dependent.
>
> Meanwhile, conda was chugging along and getting a lot of momentum in the
> Scientific community. And the core thing here is that conda was designed
> from the ground up to support essentially anything, This means is supports
> python packages that depend on non-python packages, but also supports
> packages that have nothing to do with python (Perl, command line tools,
> what have you...)
>
> So I have been focusing on conda lately.
>

The whole reason I started this discussion is to make sure wheel has a
standard way to do what is needed for those usecases.

conda, rpm, deb, or eggs as used in enthought are all essentially the same:
an archive with a bunch of metadata. The real issue is standardising on the
exact formats. As you noticed, there is not much missing in the wheel
*spec* to get most of what's needed. We've used eggs for that purpose for
almost 10 years at Enthought, and we did not need that many extensions on
top of the egg format after all.



> Which brings me back to the question: should the python tools (i.e. wheel)
> be extended to support more use-cases, specifically non-python
> dependencies? Or do we just figure that that's a problem better solved by
> projects with a larger scope (i.e. rpm, deb, conda, canopy).
>

IMO, given that wheels do most of what's needed, it is worth supporting
most simple usecases (compiled libraries required by well known
extensions). Right now, such packages (pyzmq, numpy, cryptography, lxml)
resort to quite horrible custom hacks to support those cases.

Hope that clarifies the intent,

David



> I'm on the fence here. I mostly care about Python, and I think we're
> pretty darn close with allowing wheel to support the non-python
> dependencies, which would allow us all to "simply pip install" pretty much
> anything -- that would be cool. But maybe it's a bit of a slippery slope,
> and if we go there, we'll end up re-writing conda.
>
> BTW, while you can't generally install a conda package in/for another
> python, you can generally 

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Daniel Holth]

On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau  wrote:
>
>
> On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker 
> wrote:
>>
>> NOTE: I don't work for any of the companies involved -- just a somewhat
>> frustrated user... And someone that has been trying for years to make things
>> easier for OS-X users.
>>
 I’m not sure what (3) means exactly. What is a “normal” Python, do you
 modify Python in a way that breaks the ABI but which isn’t reflected in the
 standard ABI tag?
>>>
>>>
>>> It could be multiple things. The most obvious one is that generally.
>>> cross-platforms python distributions will try to be "relocatable" (i.e. the
>>> whole installation can be moved and still work). This means they require
>>> python itself to be built a special way. Strictly speaking, it is not an ABI
>>> issue, but the result is the same though: you can't use libraries from
>>> anaconda or canopy on top of a normal python
>>
>>
>> But why not? -- at least for Anaconda, it's because those libraries likely
>> have non-python dependencies, which are expected to be installed in a
>> particular way. And really, this is not particular to Anaconda/Canopy at
>> all. Python itself has no answer for this issue, and eggs and wheels don't
>> help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to
>> build a binary wheel with non-python dependencies (let's say something like
>> libjpeg, for instance), you need to either:
>>  - assume that libjpeg is installed in a "standard" place -- really no
>> solution at all (at least outside of linux)
>>  - statically link it
>>  - ship the dynamic lib with the package
>>
>> For the most part, the accepted solution for OS-X has been to statically
>> link, but:
>>
>>  - it's a pain to do. The gnu toolchain really likes to use dynamic
>> linking, and building a static lib that will run on a
>> maybe-older-than-the-build-system machine is pretty tricky.
>>
>>  - now we end up with multiple copies of the same lib in the python
>> install. There are a handful of libs that are used a LOT. Maybe there is no
>> real downside -- disk space and memory are cheap these days, but it sure
>> feels ugly. And I have yet to feel comfortable with having multiple versions
>> of the same lib linked into one python instance -- I can't say I've seen a
>> problem, but it makes me nervous.
>>
>> On Windows, the choices are the same, except that: It is so much harder to
>> build many of the "standard" open source libs that package authors are more
>> likely to do it for folks, and you do get the occasional "dll hell" issues.
>>
>> I had a plan to make some binary wheels for OS-X that were not really
>> python packages, but actually just bundled up libs, so that other wheels
>> could depend on them. OS-X does allow linking to relative paths, so this
>> should have been doable, but I never got anyone else to agree this was a
>> good idea, and I never found the roundtoits anyway. And it doesn't really
>> fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are
>> platform dependent and even worse, build-dependent.
>>
>> Meanwhile, conda was chugging along and getting a lot of momentum in the
>> Scientific community. And the core thing here is that conda was designed
>> from the ground up to support essentially anything, This means is supports
>> python packages that depend on non-python packages, but also supports
>> packages that have nothing to do with python (Perl, command line tools, what
>> have you...)
>>
>> So I have been focusing on conda lately.
>
>
> The whole reason I started this discussion is to make sure wheel has a
> standard way to do what is needed for those usecases.
>
> conda, rpm, deb, or eggs as used in enthought are all essentially the same:
> an archive with a bunch of metadata. The real issue is standardising on the
> exact formats. As you noticed, there is not much missing in the wheel *spec*
> to get most of what's needed. We've used eggs for that purpose for almost 10
> years at Enthought, and we did not need that many extensions on top of the
> egg format after all.
>
>
>>
>> Which brings me back to the question: should the python tools (i.e. wheel)
>> be extended to support more use-cases, specifically non-python dependencies?
>> Or do we just figure that that's a problem better solved by projects with a
>> larger scope (i.e. rpm, deb, conda, canopy).
>
>
> IMO, given that wheels do most of what's needed, it is worth supporting most
> simple usecases (compiled libraries required by well known extensions).
> Right now, such packages (pyzmq, numpy, cryptography, lxml) resort to quite
> horrible custom hacks to support those cases.
>
> Hope that clarifies the intent,
>
> David

Then it sounds like I should read about the Enthought egg extensions.
It's something else than just defining a separate pypi name for "just
the libxml.so without the python bits"?
___
Distutils-SIG maillist  -  Distutils-SIG@python

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

I would advise against using or even reading about our egg extensions, as
the implementation is full of legacy (we've been doing this many years :)
): http://enstaller.readthedocs.org/en/master/reference/egg_format.html

This is what we use on top of setuptools egg:

 - ability to add dependencies which are not python packages (I think most
of it is already handled in metadata 2.0/PEP 426, but I would have to
re-read the PEP carefully).
 - ability to run post/pre install/remove scripts
 - support for all the of the autotools directories, with "sensible"
mapping on windows
 - a few extensions to the actual binary format (adding support for
symlinks is the only one I can think of ATM).

Everything else is legacy you really don't want to know (see here if you
still want to
http://enstaller.readthedocs.org/en/master/reference/egg_format.html)

David

On Mon, Apr 13, 2015 at 3:55 PM, Daniel Holth  wrote:

> On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau 
> wrote:
> >
> >
> > On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker 
> > wrote:
> >>
> >> NOTE: I don't work for any of the companies involved -- just a somewhat
> >> frustrated user... And someone that has been trying for years to make
> things
> >> easier for OS-X users.
> >>
>  I’m not sure what (3) means exactly. What is a “normal” Python, do you
>  modify Python in a way that breaks the ABI but which isn’t reflected
> in the
>  standard ABI tag?
> >>>
> >>>
> >>> It could be multiple things. The most obvious one is that generally.
> >>> cross-platforms python distributions will try to be "relocatable"
> (i.e. the
> >>> whole installation can be moved and still work). This means they
> require
> >>> python itself to be built a special way. Strictly speaking, it is not
> an ABI
> >>> issue, but the result is the same though: you can't use libraries from
> >>> anaconda or canopy on top of a normal python
> >>
> >>
> >> But why not? -- at least for Anaconda, it's because those libraries
> likely
> >> have non-python dependencies, which are expected to be installed in a
> >> particular way. And really, this is not particular to Anaconda/Canopy at
> >> all. Python itself has no answer for this issue, and eggs and wheels
> don't
> >> help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in
> order to
> >> build a binary wheel with non-python dependencies (let's say something
> like
> >> libjpeg, for instance), you need to either:
> >>  - assume that libjpeg is installed in a "standard" place -- really no
> >> solution at all (at least outside of linux)
> >>  - statically link it
> >>  - ship the dynamic lib with the package
> >>
> >> For the most part, the accepted solution for OS-X has been to statically
> >> link, but:
> >>
> >>  - it's a pain to do. The gnu toolchain really likes to use dynamic
> >> linking, and building a static lib that will run on a
> >> maybe-older-than-the-build-system machine is pretty tricky.
> >>
> >>  - now we end up with multiple copies of the same lib in the python
> >> install. There are a handful of libs that are used a LOT. Maybe there
> is no
> >> real downside -- disk space and memory are cheap these days, but it sure
> >> feels ugly. And I have yet to feel comfortable with having multiple
> versions
> >> of the same lib linked into one python instance -- I can't say I've
> seen a
> >> problem, but it makes me nervous.
> >>
> >> On Windows, the choices are the same, except that: It is so much harder
> to
> >> build many of the "standard" open source libs that package authors are
> more
> >> likely to do it for folks, and you do get the occasional "dll hell"
> issues.
> >>
> >> I had a plan to make some binary wheels for OS-X that were not really
> >> python packages, but actually just bundled up libs, so that other wheels
> >> could depend on them. OS-X does allow linking to relative paths, so this
> >> should have been doable, but I never got anyone else to agree this was a
> >> good idea, and I never found the roundtoits anyway. And it doesn't
> really
> >> fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that
> are
> >> platform dependent and even worse, build-dependent.
> >>
> >> Meanwhile, conda was chugging along and getting a lot of momentum in the
> >> Scientific community. And the core thing here is that conda was designed
> >> from the ground up to support essentially anything, This means is
> supports
> >> python packages that depend on non-python packages, but also supports
> >> packages that have nothing to do with python (Perl, command line tools,
> what
> >> have you...)
> >>
> >> So I have been focusing on conda lately.
> >
> >
> > The whole reason I started this discussion is to make sure wheel has a
> > standard way to do what is needed for those usecases.
> >
> > conda, rpm, deb, or eggs as used in enthought are all essentially the
> same:
> > an archive with a bunch of metadata. The real issue is standardising on
> the
> > exact formats. As you noticed, there is no

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Daniel Holth]

Seems like you could extend wheel to do that easily.
On Apr 13, 2015 4:19 PM, "David Cournapeau"  wrote:

> I would advise against using or even reading about our egg extensions, as
> the implementation is full of legacy (we've been doing this many years :)
> ): http://enstaller.readthedocs.org/en/master/reference/egg_format.html
>
> This is what we use on top of setuptools egg:
>
>  - ability to add dependencies which are not python packages (I think most
> of it is already handled in metadata 2.0/PEP 426, but I would have to
> re-read the PEP carefully).
>  - ability to run post/pre install/remove scripts
>  - support for all the of the autotools directories, with "sensible"
> mapping on windows
>  - a few extensions to the actual binary format (adding support for
> symlinks is the only one I can think of ATM).
>
> Everything else is legacy you really don't want to know (see here if you
> still want to
> http://enstaller.readthedocs.org/en/master/reference/egg_format.html)
>
> David
>
> On Mon, Apr 13, 2015 at 3:55 PM, Daniel Holth  wrote:
>
>> On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau 
>> wrote:
>> >
>> >
>> > On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker 
>> > wrote:
>> >>
>> >> NOTE: I don't work for any of the companies involved -- just a somewhat
>> >> frustrated user... And someone that has been trying for years to make
>> things
>> >> easier for OS-X users.
>> >>
>>  I’m not sure what (3) means exactly. What is a “normal” Python, do
>> you
>>  modify Python in a way that breaks the ABI but which isn’t reflected
>> in the
>>  standard ABI tag?
>> >>>
>> >>>
>> >>> It could be multiple things. The most obvious one is that generally.
>> >>> cross-platforms python distributions will try to be "relocatable"
>> (i.e. the
>> >>> whole installation can be moved and still work). This means they
>> require
>> >>> python itself to be built a special way. Strictly speaking, it is not
>> an ABI
>> >>> issue, but the result is the same though: you can't use libraries from
>> >>> anaconda or canopy on top of a normal python
>> >>
>> >>
>> >> But why not? -- at least for Anaconda, it's because those libraries
>> likely
>> >> have non-python dependencies, which are expected to be installed in a
>> >> particular way. And really, this is not particular to Anaconda/Canopy
>> at
>> >> all. Python itself has no answer for this issue, and eggs and wheels
>> don't
>> >> help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in
>> order to
>> >> build a binary wheel with non-python dependencies (let's say something
>> like
>> >> libjpeg, for instance), you need to either:
>> >>  - assume that libjpeg is installed in a "standard" place -- really no
>> >> solution at all (at least outside of linux)
>> >>  - statically link it
>> >>  - ship the dynamic lib with the package
>> >>
>> >> For the most part, the accepted solution for OS-X has been to
>> statically
>> >> link, but:
>> >>
>> >>  - it's a pain to do. The gnu toolchain really likes to use dynamic
>> >> linking, and building a static lib that will run on a
>> >> maybe-older-than-the-build-system machine is pretty tricky.
>> >>
>> >>  - now we end up with multiple copies of the same lib in the python
>> >> install. There are a handful of libs that are used a LOT. Maybe there
>> is no
>> >> real downside -- disk space and memory are cheap these days, but it
>> sure
>> >> feels ugly. And I have yet to feel comfortable with having multiple
>> versions
>> >> of the same lib linked into one python instance -- I can't say I've
>> seen a
>> >> problem, but it makes me nervous.
>> >>
>> >> On Windows, the choices are the same, except that: It is so much
>> harder to
>> >> build many of the "standard" open source libs that package authors are
>> more
>> >> likely to do it for folks, and you do get the occasional "dll hell"
>> issues.
>> >>
>> >> I had a plan to make some binary wheels for OS-X that were not really
>> >> python packages, but actually just bundled up libs, so that other
>> wheels
>> >> could depend on them. OS-X does allow linking to relative paths, so
>> this
>> >> should have been doable, but I never got anyone else to agree this was
>> a
>> >> good idea, and I never found the roundtoits anyway. And it doesn't
>> really
>> >> fit into the PyPi, pip, wheel, etc. philosphy to have dependencies
>> that are
>> >> platform dependent and even worse, build-dependent.
>> >>
>> >> Meanwhile, conda was chugging along and getting a lot of momentum in
>> the
>> >> Scientific community. And the core thing here is that conda was
>> designed
>> >> from the ground up to support essentially anything, This means is
>> supports
>> >> python packages that depend on non-python packages, but also supports
>> >> packages that have nothing to do with python (Perl, command line
>> tools, what
>> >> have you...)
>> >>
>> >> So I have been focusing on conda lately.
>> >
>> >
>> > The whole reason I started this discussion is to make sure wheel has a
>> > st

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Chris Barker]

On Mon, Apr 13, 2015 at 1:19 PM, David Cournapeau 
wrote:

> This is what we use on top of setuptools egg:
>
>  - ability to add dependencies which are not python packages (I think most
> of it is already handled in metadata 2.0/PEP 426, but I would have to
> re-read the PEP carefully).
>  - ability to run post/pre install/remove scripts
>  - support for all the of the autotools directories, with "sensible"
> mapping on windows
>

Are these inside or outside the python installation? I'm more than a bit
wary of a wheel that would install stuff outside of the "sandbox" of the
python install.


The whole reason I started this discussion is to make sure wheel has a
> standard way to do what is needed for those usecases.
>
> conda, rpm, deb, or eggs as used in enthought are all essentially the
> same: an archive with a bunch of metadata. The real issue is standardising
> on the exact formats. As you noticed, there is not much missing in the
> wheel *spec* to get most of what's needed.


hmm -- true. I guess where it seems to get more complicated is beyond the
wheel (or conda, or...) package itself, to the dependency management,
installation tools, etc.

But perhaps you are suggesting that we can extend wheel to support a bt
more stuff, and leave the rest of the system as separate problem? i.e.
Canopy can have it's own find, install, manage-dependency tool, but that it
can use the wheel format for the packages themselves?

I don't see why not

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

On Mon, Apr 13, 2015 at 5:25 PM, Chris Barker  wrote:

> On Mon, Apr 13, 2015 at 1:19 PM, David Cournapeau 
> wrote:
>
>> This is what we use on top of setuptools egg:
>>
>>  - ability to add dependencies which are not python packages (I think
>> most of it is already handled in metadata 2.0/PEP 426, but I would have to
>> re-read the PEP carefully).
>>  - ability to run post/pre install/remove scripts
>>  - support for all the of the autotools directories, with "sensible"
>> mapping on windows
>>
>
> Are these inside or outside the python installation? I'm more than a bit
> wary of a wheel that would install stuff outside of the "sandbox" of the
> python install.
>

I would always install things relative to sys.prefix, for exactly the
reasons you mention.


>
>
> The whole reason I started this discussion is to make sure wheel has a
>> standard way to do what is needed for those usecases.
>>
>> conda, rpm, deb, or eggs as used in enthought are all essentially the
>> same: an archive with a bunch of metadata. The real issue is standardising
>> on the exact formats. As you noticed, there is not much missing in the
>> wheel *spec* to get most of what's needed.
>
>
> hmm -- true. I guess where it seems to get more complicated is beyond the
> wheel (or conda, or...) package itself, to the dependency management,
> installation tools, etc.
>
> But perhaps you are suggesting that we can extend wheel to support a bt
> more stuff, and leave the rest of the system as separate problem? i.e.
> Canopy can have it's own find, install, manage-dependency tool, but that it
> can use the wheel format for the packages themselves?
>

Exactly !

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip/warehouse feature idea: "help needed"

[Ben Finney]

Nick Coghlan  writes:

> On 11 Apr 2015 12:22, "Alexander Walters"  wrote:
> > Is the package index really the best place to put this? This is a
> > very social-networking feature for the authoritative repository of
> > just about all the third party module, and it feels like either it
> > could corrupt the 'sanctity' of the repository (in the absolute
> > worst case)
>
> If you're concerned that this feature might weaken the comforting
> illusion that PyPI published software is contributed and maintained by
> faceless automatons rather than living, breathing human beings, then
> yes, encouraging folks to think more about where the software they use
> is coming from would be a large part of the point of adding such a
> feature.

I can't speak for Alexander, but I'm also −1 to have this *on PyPI*.

I'm all for such features existing. What is at issue is whether PyPI is
the place to put them.

We have been gradually improving the function of PyPI as an
authoritative *index* of packages; that's possible because it is a
repository of uncontroversial facts, not opinions (i.e. “what is the
packaging metadata of this distribution”, “where is its documentation”,
“where is its VCS”, etc.).

> > I am not saying the PSF shouldn't do this, but is pypi REALLY the
> > best part of python.org to put it?
>
> I personally believe so, yes - sustaining software over the long term is
> expensive in people's time, but it's often something we take for granted.
> The specific example Guido brought up in his keynote was the challenge of
> communicating a project's openness to Python 3 porting assistance.

The people doing the work of maintaining PyPI have said many times in
recent years that there just isn't enough person-power to add a whole
bunch of features that have been requested. Why would we think
moderating a social-networking rating, opinion, discussion, or other
non-factual database is something reasonable to ask of the PyPI
maintainers?

Conversely, if we are under the impression that adding ratings,
feedback, reviews, discussion, and other features to PyPI is *not* going
to be a massive increase in workload for the maintainers, I think that's
a foolish delusion which will be quite costly to the reputation PyPI has
recently gained through hard effort to clarify its role.

By all means, set up a well-maintained social ecosystem around Python
packages. But not on PyPI itself: The Python Package Index is feasible
in part because it has a clear and simple job, though, and that's not
it.

-- 
 \“If you can't hear me sometimes, it's because I'm in |
  `\  parentheses.” —Steven Wright |
_o__)  |
Ben Finney

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip/warehouse feature idea: "help needed"

[Donald Stufft]

> On Apr 13, 2015, at 8:57 PM, Ben Finney  wrote:
> 
> Nick Coghlan  writes:
> 
>> On 11 Apr 2015 12:22, "Alexander Walters"  wrote:
>>> Is the package index really the best place to put this? This is a
>>> very social-networking feature for the authoritative repository of
>>> just about all the third party module, and it feels like either it
>>> could corrupt the 'sanctity' of the repository (in the absolute
>>> worst case)
>> 
>> If you're concerned that this feature might weaken the comforting
>> illusion that PyPI published software is contributed and maintained by
>> faceless automatons rather than living, breathing human beings, then
>> yes, encouraging folks to think more about where the software they use
>> is coming from would be a large part of the point of adding such a
>> feature.
> 
> I can't speak for Alexander, but I'm also −1 to have this *on PyPI*.
> 
> I'm all for such features existing. What is at issue is whether PyPI is
> the place to put them.
> 
> We have been gradually improving the function of PyPI as an
> authoritative *index* of packages; that's possible because it is a
> repository of uncontroversial facts, not opinions (i.e. “what is the
> packaging metadata of this distribution”, “where is its documentation”,
> “where is its VCS”, etc.).
> 
>>> I am not saying the PSF shouldn't do this, but is pypi REALLY the
>>> best part of python.org to put it?
>> 
>> I personally believe so, yes - sustaining software over the long term is
>> expensive in people's time, but it's often something we take for granted.
>> The specific example Guido brought up in his keynote was the challenge of
>> communicating a project's openness to Python 3 porting assistance.
> 
> The people doing the work of maintaining PyPI have said many times in
> recent years that there just isn't enough person-power to add a whole
> bunch of features that have been requested. Why would we think
> moderating a social-networking rating, opinion, discussion, or other
> non-factual database is something reasonable to ask of the PyPI
> maintainers?
> 
> Conversely, if we are under the impression that adding ratings,
> feedback, reviews, discussion, and other features to PyPI is *not* going
> to be a massive increase in workload for the maintainers, I think that's
> a foolish delusion which will be quite costly to the reputation PyPI has
> recently gained through hard effort to clarify its role.
> 
> By all means, set up a well-maintained social ecosystem around Python
> packages. But not on PyPI itself: The Python Package Index is feasible
> in part because it has a clear and simple job, though, and that's not
> it.
> 
> --
> \“If you can't hear me sometimes, it's because I'm in |
>  `\  parentheses.” —Steven Wright |
> _o__)  |
> Ben Finney
> 
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


I don’t see any problem with the general idea of adding features to PyPI to
enable package maintainers to find more help maintaining specific parts of
their projects. I do have a problem with expecting the PyPI administrators
to fill out or otherwise populate this information. Saying “Here’s a place
you can donate to me” is still a fact, it’s just a more social fact than
what we currently enable.

I’m kind of down on the idea of linking to CVs or linkedin as part of the
project metadata because that’s not project specific and is really more
maintainer specific. I think that particular feature would be better suited
to some sort of global “Python profile” that could then be linked to from
PyPI instead of trying to bake it into PyPI itself.

However things like “Looking for New Maintainers / Orphan a Project”,
or some call to actions on “here are some issues that need fixed” or other
things doesn’t seem unreasonable to me. Particularly the ability to orphan
a project or look for new maintainers seems like a useful thing to me that
really can’t live anywhere other than PyPI reasonably.

The other items can live elsewhere if we wanted them to since they would be
easy to add to the long_description of a project which would get added to
the PyPI page but that has some drawbacks. For things like crowdfunding 
campaigns
the long_description is set when you upload a release, however it’d be useful
to have the campaigns update as the campaign progresses (or even ultimately be
removed once the campaign has finished).

I think an important part of this idea here is that this doesn’t enable anything
that authors can’t already do, it just presents the information in a way that
is easier for other tooling to take advantage of it as well as allow us to make
more informed decisions about how to show it and when to show it without 
requiring
authors to update the long_description of their projects. I think it will also
be a s

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Robert Collins]

On 14 April 2015 at 09:35, David Cournapeau  wrote:
...

One of the earlier things mentioned here - {pre,post}{install,remove}
scripts - raises a red flag for me.

In Debian at least, the underlying system has the ability to run such
turing complete scripts, and they are a rich source of bugs - both
correctness and performance related.

Nowadays nearly all such scripts are machine generated from higher
level representations such as 'this should be the default command' or
'configure X if Y is installed', but because the plumbing is turing
complete, they all need to be executed, which slows down
install/upgrade paths, and any improvement to the tooling requires a
version bump on *all* the packages using it - because effectively the
package is itself a compiled artifact.

I'd really prefer it if we keep wheels 100% declarative, and instead
focus on defining appropriate metadata for the things you need to
accomplish {pre,post}{install,remove} of a package.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Donald Stufft]

> On Apr 13, 2015, at 10:17 PM, Robert Collins  
> wrote:
> 
> On 14 April 2015 at 09:35, David Cournapeau  wrote:
> ...
> 
> One of the earlier things mentioned here - {pre,post}{install,remove}
> scripts - raises a red flag for me.
> 
> In Debian at least, the underlying system has the ability to run such
> turing complete scripts, and they are a rich source of bugs - both
> correctness and performance related.
> 
> Nowadays nearly all such scripts are machine generated from higher
> level representations such as 'this should be the default command' or
> 'configure X if Y is installed', but because the plumbing is turing
> complete, they all need to be executed, which slows down
> install/upgrade paths, and any improvement to the tooling requires a
> version bump on *all* the packages using it - because effectively the
> package is itself a compiled artifact.
> 
> I'd really prefer it if we keep wheels 100% declarative, and instead
> focus on defining appropriate metadata for the things you need to
> accomplish {pre,post}{install,remove} of a package.


A possible way to implement {pre,post}{install,remove} scripts is to
instead turn them into extensions. One example is that Twisted uses
a setup.py hack to regenerate a cache file of all of the registered
plugins. This needs to happen at install time due to permission issues.
Currently you can’t get this speedup when installing something that
uses twisted plugins and installing via Wheel.

So a possible way for this to work is in a PEP 426 world, simply define
a twisted.plugins extension that says, in a declarative way, “hey when
you install this Wheel, if there’s a plugin that understands this extension
installed, let it do something before you actually move the files into
place”. This let’s Wheels themselves still be declarative and moves the
responsibility of implementing these bits into their own PyPI projects
that can be versioned and independently upgraded and such. We’d probably
need some method of marking an extension as “critical” (e.g. bail out and
don’t install this Wheel if you don’t have something that knows how to handle
it) and then non critical extensions just get ignored if we don’t know
how to handle it.

Popular extensions could possibly be added directly to pip at some point
if a lot of people are using them (or even moved from third party extension
to officially supported extension).

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[David Cournapeau]

On Mon, Apr 13, 2015 at 10:17 PM, Robert Collins 
wrote:

> On 14 April 2015 at 09:35, David Cournapeau  wrote:
> ...
>
> One of the earlier things mentioned here - {pre,post}{install,remove}
> scripts - raises a red flag for me.
>

That's indeed a good a priori. I myself removed a lot of those scripts
because of the fragility. Anything that needs to run on a end-user machine
can fail, and writing idempotent scripts is hard.

Unfortunately, pure declarative does not really cut it if you want cross
platform support. Sure, you may be able to deal with menu entries,
environment variables, etc... in a cross-platform manner with a significant
effort, but what about COM registration ? pywin32 is one of the most used
package in the python ecosystem, and its post install script is not
trivial. Another difficult case if when a package needs some specific
configuration to run at all, and that configuration requires values known
at install time only (e.g. sys.prefix, as in the iris package).


> I'd really prefer it if we keep wheels 100% declarative, and instead
> focus on defining appropriate metadata for the things you need to
> accomplish {pre,post}{install,remove} of a package.
>

What about a way for wheels to specify whether their
post,pre/install,remove scripts are declarative or not, with support for
the most common tasks, with an escape, but opt-in mechanism ? This way it
could be a matter of policy to refuse packages that require non-declarative
scripts.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Paul Moore]

On 14 April 2015 at 06:37, David Cournapeau  wrote:
> pywin32 is one of the most used package in the python ecosystem, and its
> post install script is not trivial.

And yet pywin32's postinstall script is completely virtualenv-hostile.
It registers start menu entries (not applicable when installing in a
virtualenv), registers itself as a COM server (once again, one per
machine), adds registry entries (again, virtualenv-hostile), moves
installed files into the Windows system directory (ditto) etc. And yet
for many actual uses of pywin32, installing as a wheel without running
the postinstall is sufficient. With the exception of writing COM
servers in Python (and maybe writing services, but I thing cx_Freeze
lets you do that without pywin32), pretty much every use *I* have seen
of pywin32 can be replaced with ctypes or cffi with no loss of
functionality.

I'd argue that pywin32 is a perfect example of a project where *not*
supporting postinstall scripts would be a good idea, as it would
encourage the project to find a way to implement the same
functionality in a way that's compatible with current practices
(virtualenv, tox, etc). Or it would encourage other projects to stop
depending on pywin32 (which is actually what is happening, many
projects now use ctypes and similar in place of pywin32-using code, to
avoid the problems pywin32 causes for them).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig