[Distutils] elements of setuptools/wheel/FHS file categories proposal

[Daniel Holth]

It might be helpful to discuss the fine grained install scheme
proposal as several proposals:

- Add more install paths (categories) to the set currently used by
distutils/sysconfig
https://docs.python.org/3/library/sysconfig.html#installation-paths .
Define sensible defaults for each platform.

- Allow the paths to be individually overridden for each installed
package, in the same way that python setup.py install
--install-purelib=/some-directory can override the categories we have
now.

- Record the { category : path } mapping used during installation.

- Provide an API mapping (distribution name, category, relative path
within category) to help applications using data that is only
accessible via the module loader.
https://docs.python.org/3/library/importlib.html#importlib.abc.ResourceLoader

- Provide an API mapping the same to paths on the filesystem.

- Make the recorded mapping available in a predictable location, so it
can be perhaps understood by non-Python code.

- Allow setup.py's setup() call to install files relative to each
defined category.

- Extend Python binary package formats to support the new categories,
so the { category : path } mapping can be set at install time and not
at build time.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[Steve Dower]

My brief POV is that if a package on Windows is installing anything outside 
sys.path at all then it's an application and should use something other than 
wheel for installation. WiX/MSI will do proper reference counting and upgrades 
to avoid having multiple versions colliding with each other (imagine installing 
Mercurial on 2.6 and 2.7 simultaneously...)

These should probably also bundle the interpreter as well to avoid other update 
issues. I'm already going to release a self extracting zip of the interpreter 
for this, though there will be tweaks necessary to the initialization process 
to avoid registry stomping, and tools like pynsist are being worked on to 
simply installer generation. These can freely install files anywhere, and 
having a consistent spec for generating these would be nice, but I don't want 
pip to install files outside of sys.path

Cheers,
Steve

Top-posted from my Windows Phone

From: Nick Coghlan
Sent: ‎4/‎19/‎2015 9:41
To: Paul Moore
Cc: DistUtils mailing list
Subject: Re: [Distutils] Add additional file categories for distutils, 
setuptools, wheel

On 19 April 2015 at 06:03, Paul Moore  wrote:
> As a possible compromise, how about an approach where on Linux system
> installs (or more accurately "those install schemes that are relevant
> to the distribution packaging issue you described") the file
> categories are installed into dedicated directories, as described. But
> on other installs (virtualenvs, Windows, maybe OSX) the file
> categories map to locations within package_data, so that they can be
> accessed via normal mechanisms like loader.get_data. Application code
> would need some support code to locate and read the files, but that's
> true whether we go for this proposal or an "outside of site-packages"
> scheme. Also, some things may even be better designated as "don't
> install" in certain schemes (man files, for example, are pretty much
> useless on Windows).

That's not a compromise, that's exactly what I want to see happen :)

If it helps, one way to think of this is as a "file classification"
system, where the packaging tools gain the ability to sort files into
a richer set of categories, and its then up to installers to decide
how those categories map to installation locations. For Windows,
virtualenv, conda, nix, single file applications, etc, that's likely
to be "self-contained application directory". For FHS, it will map to
the on-disk categorisation expected by other tools.

At the moment, because the FHS support is either hacked in, or managed
externally, there's no way for installers to remap the installation
targets for the "self-contained directory" use case.

Cheers,
Nick.

--
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[David Cournapeau]

On Sun, Apr 19, 2015 at 6:39 PM, Vinay Sajip 
wrote:

>
>
>
>
> From: Paul Moore 
> > On 19 April 2015 at 11:55, David Cournapeau  wrote:
> >> I can work on a "paperman" implementation of this on top of the "wheel"
> >> installer for the end of this week. I think that would both alleviate
> some
> >> concerns for people interested in "everything in package directory", and
> >> make the discussion more focused.
>
> Is "paperman" a Disney reference, or something else?
>

it was used in another project I was involved in, to mean "even less than a
straw man". Since the project was handled by native speakers, I assumed it
was a generic term :)

In any case, the implementation I started in wheel was something to *start*
the discussion, not to be used as a reference.

David


>
> > That sounds good. One thing the wheel install command doesn't support
>
> > is per-user installs. I'd appreciate seeing some details of how those
> > would be handled - on the assumption that you don't want to deal with
> > pip's source just yet, a rough spec would be fine for now.
> I presume the way "wheel install" works is orthogonal to the scheme for
> handling different categories of data - "distil install", for example, does
> per-user installs by default. Are the proposed implementations just a proof
> of concept, to validate the usability of the implemented scheme? What is
> the envisaged timeline for proposing/agreeing specifications for how the
> file categories will work cross-platform? I'm bearing in mind that there
> might be other implementations of installers which would need to
> interoperate with any file category scheme ...
>
> Regards,
>
> Vinay Sajip
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[Paul Moore]

On 19 April 2015 at 17:41, Nick Coghlan  wrote:
> On 19 April 2015 at 06:03, Paul Moore  wrote:
>> As a possible compromise, how about an approach where on Linux system
>> installs (or more accurately "those install schemes that are relevant
>> to the distribution packaging issue you described") the file
>> categories are installed into dedicated directories, as described. But
>> on other installs (virtualenvs, Windows, maybe OSX) the file
>> categories map to locations within package_data, so that they can be
>> accessed via normal mechanisms like loader.get_data. Application code
>> would need some support code to locate and read the files, but that's
>> true whether we go for this proposal or an "outside of site-packages"
>> scheme. Also, some things may even be better designated as "don't
>> install" in certain schemes (man files, for example, are pretty much
>> useless on Windows).
>
> That's not a compromise, that's exactly what I want to see happen :)
>
> If it helps, one way to think of this is as a "file classification"
> system, where the packaging tools gain the ability to sort files into
> a richer set of categories, and its then up to installers to decide
> how those categories map to installation locations. For Windows,
> virtualenv, conda, nix, single file applications, etc, that's likely
> to be "self-contained application directory". For FHS, it will map to
> the on-disk categorisation expected by other tools.
>
> At the moment, because the FHS support is either hacked in, or managed
> externally, there's no way for installers to remap the installation
> targets for the "self-contained directory" use case.

OK, that's fantastic. It sounds much more reasonable when put that
way. The only big debate then is likely to be over the precise
categories used. And in particular, should we simply take an existing
Unix-centric set of categories like the autotools ones, as currently
proposed, or should we choose something more cross-platform?

In favour of Unix autotools-style classes:

1. Already exists, easy to just take what's already defined.
2. Proven utility (although possibly over-engineered, some of the
autotools categories seem pretty obscure to my admittedly untrained
eye).
3. Mapping to Unix systems (and hence distribution packaging systems)
is obvious.
4. Generally, Windows users don't care about any of this as the model
on Windows is to install everything in the application directory, so
following Unix conventions will be less controversial.

Against:

1. The categories are pretty meaningless for Windows developers, so
would tend to be ignored or used incorrectly by them.
2. Makes it too easy for Unix users to ignore or misunderstand
cross-platform issues (for example, the mandir category implies that
shipping man pages is a reasonable way of documenting your package,
but Windows users won't be able to read the manpages, and they may not
even get installed on Windows).
3. Easy to interpret as "treating Windows as a second class citizen".

Overall, I don't see enough disadvantages in there to argue against
the autotools classifications, but I would like to see the PEP include
a discussion of the portability implications of the new feature.

And there are still a load of questions waiting on a clear spec of how
installers are expected to map the categories to filesystem locations,
for all the various OS/install scheme combinations that exist. But
it's not worth speculating on those, better to wait for the spec and
then discuss the details.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[Daniel Holth]

He probably means straw man, or an implementation built only to facilitate
discussion.

I'd like to get it done before October.

Ideally the installs should still work regardless of exactly where each
category maps on disk.
On Apr 19, 2015 12:43 PM, "Vinay Sajip"  wrote:

>
>
>
>
> From: Paul Moore 
> > On 19 April 2015 at 11:55, David Cournapeau  wrote:
> >> I can work on a "paperman" implementation of this on top of the "wheel"
> >> installer for the end of this week. I think that would both alleviate
> some
> >> concerns for people interested in "everything in package directory", and
> >> make the discussion more focused.
>
> Is "paperman" a Disney reference, or something else?
>
> > That sounds good. One thing the wheel install command doesn't support
>
> > is per-user installs. I'd appreciate seeing some details of how those
> > would be handled - on the assumption that you don't want to deal with
> > pip's source just yet, a rough spec would be fine for now.
> I presume the way "wheel install" works is orthogonal to the scheme for
> handling different categories of data - "distil install", for example, does
> per-user installs by default. Are the proposed implementations just a proof
> of concept, to validate the usability of the implemented scheme? What is
> the envisaged timeline for proposing/agreeing specifications for how the
> file categories will work cross-platform? I'm bearing in mind that there
> might be other implementations of installers which would need to
> interoperate with any file category scheme ...
>
> Regards,
>
> Vinay Sajip
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[Vinay Sajip]

From: Paul Moore 
> On 19 April 2015 at 11:55, David Cournapeau  wrote:
>> I can work on a "paperman" implementation of this on top of the "wheel"
>> installer for the end of this week. I think that would both alleviate some
>> concerns for people interested in "everything in package directory", and
>> make the discussion more focused.

Is "paperman" a Disney reference, or something else?

> That sounds good. One thing the wheel install command doesn't support

> is per-user installs. I'd appreciate seeing some details of how those
> would be handled - on the assumption that you don't want to deal with
> pip's source just yet, a rough spec would be fine for now.
I presume the way "wheel install" works is orthogonal to the scheme for 
handling different categories of data - "distil install", for example, does 
per-user installs by default. Are the proposed implementations just a proof of 
concept, to validate the usability of the implemented scheme? What is the 
envisaged timeline for proposing/agreeing specifications for how the file 
categories will work cross-platform? I'm bearing in mind that there might be 
other implementations of installers which would need to interoperate with any 
file category scheme ...

Regards,

Vinay Sajip
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[Nick Coghlan]

On 19 April 2015 at 06:03, Paul Moore  wrote:
> As a possible compromise, how about an approach where on Linux system
> installs (or more accurately "those install schemes that are relevant
> to the distribution packaging issue you described") the file
> categories are installed into dedicated directories, as described. But
> on other installs (virtualenvs, Windows, maybe OSX) the file
> categories map to locations within package_data, so that they can be
> accessed via normal mechanisms like loader.get_data. Application code
> would need some support code to locate and read the files, but that's
> true whether we go for this proposal or an "outside of site-packages"
> scheme. Also, some things may even be better designated as "don't
> install" in certain schemes (man files, for example, are pretty much
> useless on Windows).

That's not a compromise, that's exactly what I want to see happen :)

If it helps, one way to think of this is as a "file classification"
system, where the packaging tools gain the ability to sort files into
a richer set of categories, and its then up to installers to decide
how those categories map to installation locations. For Windows,
virtualenv, conda, nix, single file applications, etc, that's likely
to be "self-contained application directory". For FHS, it will map to
the on-disk categorisation expected by other tools.

At the moment, because the FHS support is either hacked in, or managed
externally, there's no way for installers to remap the installation
targets for the "self-contained directory" use case.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Idea: move accepted interoperability specifications to packaging.python.org

[Marcus Smith]

the PyPA site has a PEP reference that includes details on implementation:

https://www.pypa.io/en/latest/peps

I don't think we need another reference in the Packaging User Guide
(PUG).   We could mention that the PyPA one exists in the PUG.

As for user-facing PEP docs, I think the docs for PUG/pip/setuptools/wheel
etc.. should handle that inline, and act as the layer that references (if
need be) the correct PEP that applies to the feature being described.

For example, the PUG references PEP440 in a few places.

Ideally, it should be a requirement to update the docs for the major
projects (and the PUG) *before* releasing a PEP-implementations into those
projects.

For example, a round of docs PRs prior to the release of PEP440 into pug &
pip  could have likely prevented the confusion over the meaning  of ">"
that unfortunately had to be sorted out after the release.


On Sun, Apr 19, 2015 at 1:21 AM, holger krekel  wrote:

>
> I'd appreciate a "current packaging specs" site which ideally also states
> how pypa tools support it, since which version.
>
> holger
>
> On Fri, Apr 17, 2015 at 16:18 -0400, Nick Coghlan wrote:
> > Daniel's started work on a new revision of the wheel specification,
> > and it's crystallised a concern for me that's been building for a
> > while: the Python Enhancement Proposal process is fundamentally a
> > *change management* process and fundamentally ill-suited to acting as
> > a *hosting service* for the resulting reference documentation.
> >
> > This is why we're seeing awkward splits like the one I have in PEP
> > 440, where the specification itself is up top, with the rationale for
> > changes below, and the large amounts of supporting material in PEP
> > 426, where the specification is mixed in with a lot of background and
> > rationale that isn't relevant if you just want the technical details
> > of the latest version of the format.
> >
> > It also creates a problem where links to PEP based reference documents
> > are fundamentally unstable - when we define a new version of the wheel
> > format in a new PEP then folks are going to have to follow the daisy
> > chain from PEP 427 through to the new PEP, rather than having a stable
> > link that automatically presents the latest version of the format,
> > perhaps with archived copies of the old version readily available.
> >
> > I think I see a way to resolve this, and I believe it should be fairly
> > straightforward: we could create a "specifications" section on
> > packaging.python.org, and as we next revise them, we start migrating
> > the specs themselves out of the PEP system and into
> > packaging.python.org. This would be akin to the change in the Python
> > 3.3, where the specification of the way the import system worked
> > finally moved from PEP 302 into the language reference.
> >
> > Under that model, the wheel 2.0 would be specifically focused on
> > describing and justifying the *changes* between 1.0 and 2.0, but the
> > final spec itself would be a standalone document living on
> > packaging.python.org, and prominently linked to from both PEP 427
> > (which it would Supersede) and from the new PEP.
> >
> > This approach also gives a much nicer model for fixing typos in the
> > specifications - those would just be ordinary GitHub PR's on the
> > packaging.python.org repo, rather than needing to update the PEPs
> > repo.
> >
> > Thoughts?
> >
> > Regards,
> > Nick.
> >
> > --
> > Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
>
> --
> about me:http://holgerkrekel.net/about-me/
> contracting: http://merlinux.eu
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Idea: move accepted interoperability specifications to packaging.python.org

[Nick Coghlan]

On 19 April 2015 at 04:21, holger krekel  wrote:
>
> I'd appreciate a "current packaging specs" site which ideally also states
> how pypa tools support it, since which version.

OK, I've filed this idea as
https://github.com/pypa/python-packaging-user-guide/issues/151, but I
have no idea when I'll find time to work on it myself. If anyone had
the time and inclination to start putting something together (perhaps
starting with the already accepted PEP 440), that would be wonderful.

Cheers,
Nick.


>
> holger
>
> On Fri, Apr 17, 2015 at 16:18 -0400, Nick Coghlan wrote:
>> Daniel's started work on a new revision of the wheel specification,
>> and it's crystallised a concern for me that's been building for a
>> while: the Python Enhancement Proposal process is fundamentally a
>> *change management* process and fundamentally ill-suited to acting as
>> a *hosting service* for the resulting reference documentation.
>>
>> This is why we're seeing awkward splits like the one I have in PEP
>> 440, where the specification itself is up top, with the rationale for
>> changes below, and the large amounts of supporting material in PEP
>> 426, where the specification is mixed in with a lot of background and
>> rationale that isn't relevant if you just want the technical details
>> of the latest version of the format.
>>
>> It also creates a problem where links to PEP based reference documents
>> are fundamentally unstable - when we define a new version of the wheel
>> format in a new PEP then folks are going to have to follow the daisy
>> chain from PEP 427 through to the new PEP, rather than having a stable
>> link that automatically presents the latest version of the format,
>> perhaps with archived copies of the old version readily available.
>>
>> I think I see a way to resolve this, and I believe it should be fairly
>> straightforward: we could create a "specifications" section on
>> packaging.python.org, and as we next revise them, we start migrating
>> the specs themselves out of the PEP system and into
>> packaging.python.org. This would be akin to the change in the Python
>> 3.3, where the specification of the way the import system worked
>> finally moved from PEP 302 into the language reference.
>>
>> Under that model, the wheel 2.0 would be specifically focused on
>> describing and justifying the *changes* between 1.0 and 2.0, but the
>> final spec itself would be a standalone document living on
>> packaging.python.org, and prominently linked to from both PEP 427
>> (which it would Supersede) and from the new PEP.
>>
>> This approach also gives a much nicer model for fixing typos in the
>> specifications - those would just be ordinary GitHub PR's on the
>> packaging.python.org repo, rather than needing to update the PEPs
>> repo.
>>
>> Thoughts?
>>
>> Regards,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
> --
> about me:http://holgerkrekel.net/about-me/
> contracting: http://merlinux.eu



-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Secondary package indexes

[Paul Moore]

On 19 April 2015 at 14:49, Ionel Cristian Mărieș  wrote:
>
> On Sun, Apr 19, 2015 at 4:10 PM, Paul Moore  wrote:
>>
>> you can add an
>> extra_index_url to your pip.ini
>
> Would this work as expected if a package is in multiple indexes? Eg: sdist
> in main index, wheel in extra index.

Yes.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Secondary package indexes

[Ionel Cristian Mărieș]

On Sun, Apr 19, 2015 at 4:10 PM, Paul Moore  wrote:

> you can add an
> extra_index_url to your pip.ini
>

​Would this work as expected if a package is in multiple indexes? Eg: sdist
in main index, wheel in ​extra index.



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Secondary package indexes

[Paul Moore]

On 19 April 2015 at 14:14, Ionel Cristian Mărieș  wrote:
> So what you're saying is that Christoph Gohlke could use binstar to host the
> wheels yes?

Possibly. It depends on whether his use conformed to the free plans,
or he wanted to take out a paid plan. I thought you were talking about
making your own wheels.
Paul.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Secondary package indexes

[Ionel Cristian Mărieș]

So what you're saying is that Christoph Gohlke could use binstar to host
the wheels yes?


Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro

On Sun, Apr 19, 2015 at 4:10 PM, Paul Moore  wrote:

> On 19 April 2015 at 14:00, Ionel Cristian Mărieș 
> wrote:
> > Probably this has been discussed in the past but I'm asking anyway cause
> I'm
> > not sure what's it at now.
> >
> > Currently there's this problem with wheels, many package authors don't
> > publish them for the platforms I'm using. I'm speaking about the wheels
> that
> > need a compiler and/or other annoying dependencies.
> >
> > It would be really nice if one could configure pip to look into a
> secondary
> > (external) indexes. This way someone could make his own index with
> windows
> > wheels, another person could make an index with wheels for ubuntu 14.04
> and
> > people can use those to avoid the pain of compiling the packages.
> >
> > This is already popular, eg: http://www.lfd.uci.edu/~gohlke/pythonlibs/
> but
> > not readily usable with pip.
> >
> > Maybe PyPI could even provide hosting for these 3rd party wheels, this
> would
> > make publishing very easy and lower the entry bar.
> >
> > I believe we'll never get package authors to publish wheels for platforms
> > they don't care about. But other people might - we just need to make this
> > easy and convenient.
>
> I'm not sure exactly what you're suggesting here, but you can add an
> extra_index_url to your pip.ini. As far as hosting is concerned, if
> you don't want to set up your own package index, binstar offers
> pypi-style hosting for wheels. I'm not sure what reasons there may be
> for Christoph Gohlke not hosting his wheels in a pip-usable format. It
> may be worth someone asking him. I know it'd be nicer for me to point
> to his index, rather than downloading the wheels I need and hosting
> them myself for my personal use.
>
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Secondary package indexes

[Paul Moore]

On 19 April 2015 at 14:00, Ionel Cristian Mărieș  wrote:
> Probably this has been discussed in the past but I'm asking anyway cause I'm
> not sure what's it at now.
>
> Currently there's this problem with wheels, many package authors don't
> publish them for the platforms I'm using. I'm speaking about the wheels that
> need a compiler and/or other annoying dependencies.
>
> It would be really nice if one could configure pip to look into a secondary
> (external) indexes. This way someone could make his own index with windows
> wheels, another person could make an index with wheels for ubuntu 14.04 and
> people can use those to avoid the pain of compiling the packages.
>
> This is already popular, eg: http://www.lfd.uci.edu/~gohlke/pythonlibs/ but
> not readily usable with pip.
>
> Maybe PyPI could even provide hosting for these 3rd party wheels, this would
> make publishing very easy and lower the entry bar.
>
> I believe we'll never get package authors to publish wheels for platforms
> they don't care about. But other people might - we just need to make this
> easy and convenient.

I'm not sure exactly what you're suggesting here, but you can add an
extra_index_url to your pip.ini. As far as hosting is concerned, if
you don't want to set up your own package index, binstar offers
pypi-style hosting for wheels. I'm not sure what reasons there may be
for Christoph Gohlke not hosting his wheels in a pip-usable format. It
may be worth someone asking him. I know it'd be nicer for me to point
to his index, rather than downloading the wheels I need and hosting
them myself for my personal use.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Secondary package indexes

[Ionel Cristian Mărieș]

Hello,

Probably this has been discussed in the past but I'm asking anyway cause
I'm not sure what's it at now.

Currently there's this problem with wheels, many package authors don't
publish them for the platforms I'm using. I'm speaking about the wheels
that need a compiler and/or other annoying dependencies.

It would be really nice if one could configure pip to look into a secondary
(external) indexes. This way someone could make his own index with windows
wheels, another person could make an index with wheels for ubuntu 14.04 and
people can use those to avoid the pain of compiling the packages.

This is already popular, eg: http://www.lfd.uci.edu/~gohlke/pythonlibs/ but
not readily usable with pip.

Maybe PyPI could even provide hosting for these 3rd party wheels, this
would make publishing very easy and lower the entry bar.

I believe we'll never get package authors to publish wheels for platforms
they don't care about. But other people might - we just need to make this
easy and convenient.

Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[Paul Moore]

On 19 April 2015 at 11:55, David Cournapeau  wrote:
> I can work on a "paperman" implementation of this on top of the "wheel"
> installer for the end of this week. I think that would both alleviate some
> concerns for people interested in "everything in package directory", and
> make the discussion more focused.

That sounds good. One thing the wheel install command doesn't support
is per-user installs. I'd appreciate seeing some details of how those
would be handled - on the assumption that you don't want to deal with
pip's source just yet, a rough spec would be fine for now.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[David Cournapeau]

Thanks for pushing this Daniel.

I think we should wait for a bit before making this proposal official. Just
after I made my PR against wheel to implement this, I had some discussions
with Nathaniel Smith from numpy, where he remarked we may want to support
better "everything in site-packages" model.

At the lowest level, the supported schema supports both usecases. We know
that because distributions like NixOS support the "one directory per
package" model even though most packages they package use autotools scheme.

But we may want to add higher level support at the same time as this new
scheme to avoid people coming up with their own custom solutions. One idea
that was thrown out was enabling a pkg-config-like mechanism to separate
where files are from how to find information for building things. That
would allow inside site-packages and outside site-packages schemes to work
seamlessly.

I can work on a "paperman" implementation of this on top of the "wheel"
installer for the end of this week. I think that would both alleviate some
concerns for people interested in "everything in package directory", and
make the discussion more focused.

On Sat, Apr 18, 2015 at 11:24 PM, Daniel Holth  wrote:

> I am working on a minor update to the wheel format to add more
> categories under which files can be installed. Constructive comments
> welcome.
>
> Distutills, setuptools, wheel currently have the best support for
> installing files relative to ('purelib', 'platlib', 'headers',
> 'scripts', 'data') with 'data' usually meaning '/' or the root of the
> virtualenv.  In practice only exactly one of the 'purelib' or
> 'platlib' locations (which are usually mapped to the same directory on
> disk), and sometimes 'scripts' is used for any given package, and
> Python packages have no good way of loading any of their package files
> at run time if they are installed into any location not relative to
> sys.path.
>
> This works fairly well for Python libraries, but anyone packaging
> applications for a Linux distribution is required to follow the
> filesystem hierarchy standard or FHS.
>
>
> http://docs.fedoraproject.org/en-US/Fedora/14/html/Storage_Administration_Guide/s1-filesystem-fhs.html
> ,
> http://www.debian.org/releases/etch/amd64/apcs02.html,
> http://www.pathname.com/fhs/
>
> It would help Linux distribution package maintainers and Python
> application (not library) developers if wheel had better support for
> installing files into the FHS, but it would help everyone else who
> wanted to generate cross-platform packages if the FHS was not
> hardcoded as paths in the data/ category. To that end we should come
> up with some additional categories that map, or do not map, to the FHS
> based on the end user's platform.
>
> Distutils2 had the concept of resource categories:
> http://alexis.notmyidea.org/distutils2/setupcfg.html#resources
>
> """
> Default categories are:
>
> * config
> * appdata
> * appdata.arch
> * appdata.persistent
> * appdata.disposable
> * help
> * icon
> * scripts
> * doc
> * info
> * man
> """
>
> GNU has directory variables:
> https://www.gnu.org/prep/standards/html_node/Directory-Variables.html
> , for example prefix, exec_prefix, bindir, sbindir.
>
> Bento has a list based on the GNU paths, and allows new paths to be
> defined:
>
> prefix: install architecture-independent files
> eprefix: install architecture-dependent files
> bindir: user executables
> sbindir: system admin executables
> libexecdir: program executables
> sysconfdir: read-only single-machine data
> sharedstatedir: modifiable architecture-independent data
> localstatedir: modifiable single-machine data
> libdir: object code libraries
> includedir: C header files
> oldincludedir: C header files for non-gcc
> datarootdir: read-only arch.-independent data root
> datadir: read-only architecture-independent data
> infodir: info documentation
> localedir: locale-dependent data
> mandir: man documentation
> docdir: documentation root
> htmldir: html documentation
> dvidir: dvi documentation
> pdfdir: pdf documentation
> psdir: ps documentation
>
> I would like to add Bento's list to wheel and to setuptools. We would
> fix the disused setup.py setup(data_files = ) argument so that it
> could be used with $ substitution, or provide a utility function that
> would expand them in setup.py itself:
>
> data_files = { '$libdir/mylib' : [ 'some_library_file'],
> '$datadir/mydata' : ['some_data_file']}
>
> We would provide a default configuration file that mapped the
> categories to their installed locations.
>
> We would store the actual paths used at install time, so a package
> could look for files relative to the $datadir used when it was
> installed.
>
> Then it would be easier to distribute Python programs that need to
> install some files to paths that are not relative to the site-packages
> directory.
>
> - Daniel Holth
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://ma

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[Paul Moore]

On 19 April 2015 at 04:42, Nick Coghlan  wrote:
> On 18 April 2015 at 21:31, Daniel Holth  wrote:
>> Remember when distutils-sig didn't believe that Windows users actually
>> lacked compilers?

:-) I remember the days before we had package_data, when projects had
to abuse the data file support in setuptools to create $prefix/share,
or $prefix/docs. Projects like moin and cx_Oracle did this, for
example. It looked a mess on Windows installations :-)

>> graphite-web and carbon are two examples of Python applications with
>> lots of data files, that want to install config files into an obvious
>> place so you can edit them and get the applications up and running.
>> Because Python packaging has such terrible support for applications
>> that are not libraries, it's unnecessarily hard to get these packages
>> up and running -- especially if you want to put them in a virtualenv.

Thanks. If I get the time, I'll have a look at those packages and see
how their requirements might translate to Windows.

> It's a fairly standard practice in Fedora/RHEL/CentOS to use setup.py
> to define the build process in the RPM spec file, even if the package
> itself is never distributed via the upstream package index (e.g.
> beaker-project.org is built that way, as is pulpproject.org).
>
> Fedora's packaging policy for redistribution of upstream Python
> projects also switched last year to favour using "pip install" in the
> RPM build process over invoking "setup.py install" directly.
>
> Historically, all the "extra bits" needed to comply with FHS have
> lived in the spec file, independently of the upstream packaging
> system, requiring changes in two places for certain kinds of packaging
> modifications, and frequently rendering the projects undeployable in a
> virtual environment for no good reason.
>
> The benefit of Daniel's proposal is that it should make it feasible to
> modify many of these projects to be virtualenv friendly, and then
> *automate* the process of generating FHS policy compliant downstream
> packages. That will be a big step towards "package for PyPI, get your
> conda/Nix/Debian/Fedora packaging for free", so it feeds directly into
> my own interests in streamlining the redistribution pipeline in
> Fedora.

OK, thanks. That clarifies why this is needed, but it does read to me
as being something that projects should use sparingly, and only when
they are facing the sort of Linux packaging issues that it's designed
to solve. That's fine - my worry is that it'll be easy for projects to
see it as the "normal" way to structure their data, and as a result
we'll see a move away from package_data (which was introduced
precisely to solve the sorts of problems that the Unix model of
dedicated directories introduced in the context of Windows, zipimport
and virtualenv).

> From a Windows perspective, I believe this change mostly has the
> potential to make services that were previously Linux-only solely for
> packaging related reasons available on Windows as well. However, there
> may also be an opportunity to better automate the process of
> generating wix-based installers from PyPI packages (see
> http://wixtoolset.org/documentation/manual/v3/howtos/files_and_registry/add_a_file.html)
> rather than generating Windows installers directly (if I understand
> the tooling correctly, introducing wix into that process should offer
> the same kind of potential for better platform integration that
> integrating with distro package managers offers on Linux).

The biggest problem I see from a Windows POV is that installing files
outside of the package data structure is actively hostile to tools
like py2exe, and cx_Freeze, which are widely used for bundling
standalone Windows applications written in Python. Such tools won't be
able to find data installed via the proposed file category mechanism.

As a possible compromise, how about an approach where on Linux system
installs (or more accurately "those install schemes that are relevant
to the distribution packaging issue you described") the file
categories are installed into dedicated directories, as described. But
on other installs (virtualenvs, Windows, maybe OSX) the file
categories map to locations within package_data, so that they can be
accessed via normal mechanisms like loader.get_data. Application code
would need some support code to locate and read the files, but that's
true whether we go for this proposal or an "outside of site-packages"
scheme. Also, some things may even be better designated as "don't
install" in certain schemes (man files, for example, are pretty much
useless on Windows).

Beyond this issue, I do have some concerns over the specific locations
proposed, but they are better addressed as part of the PEP review
process, once we have a specific proposal to review.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Idea: move accepted interoperability specifications to packaging.python.org

[holger krekel]

I'd appreciate a "current packaging specs" site which ideally also states
how pypa tools support it, since which version.

holger

On Fri, Apr 17, 2015 at 16:18 -0400, Nick Coghlan wrote:
> Daniel's started work on a new revision of the wheel specification,
> and it's crystallised a concern for me that's been building for a
> while: the Python Enhancement Proposal process is fundamentally a
> *change management* process and fundamentally ill-suited to acting as
> a *hosting service* for the resulting reference documentation.
> 
> This is why we're seeing awkward splits like the one I have in PEP
> 440, where the specification itself is up top, with the rationale for
> changes below, and the large amounts of supporting material in PEP
> 426, where the specification is mixed in with a lot of background and
> rationale that isn't relevant if you just want the technical details
> of the latest version of the format.
> 
> It also creates a problem where links to PEP based reference documents
> are fundamentally unstable - when we define a new version of the wheel
> format in a new PEP then folks are going to have to follow the daisy
> chain from PEP 427 through to the new PEP, rather than having a stable
> link that automatically presents the latest version of the format,
> perhaps with archived copies of the old version readily available.
> 
> I think I see a way to resolve this, and I believe it should be fairly
> straightforward: we could create a "specifications" section on
> packaging.python.org, and as we next revise them, we start migrating
> the specs themselves out of the PEP system and into
> packaging.python.org. This would be akin to the change in the Python
> 3.3, where the specification of the way the import system worked
> finally moved from PEP 302 into the language reference.
> 
> Under that model, the wheel 2.0 would be specifically focused on
> describing and justifying the *changes* between 1.0 and 2.0, but the
> final spec itself would be a standalone document living on
> packaging.python.org, and prominently linked to from both PEP 427
> (which it would Supersede) and from the new PEP.
> 
> This approach also gives a much nicer model for fixing typos in the
> specifications - those would just be ordinary GitHub PR's on the
> packaging.python.org repo, rather than needing to update the PEPs
> repo.
> 
> Thoughts?
> 
> Regards,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
> 

-- 
about me:http://holgerkrekel.net/about-me/
contracting: http://merlinux.eu
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig