Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Marcus Smith]

PEP number yet?

On Sun, Nov 22, 2015 at 4:45 PM, Donald Stufft  wrote:

> Okay. I’ve read over this, implemented enough of it, and I think it’s gone
> through enough nit picking. I’m going to go ahead and accept this PEP. It’s
> largely just standardizing what we are already doing so it’s pretty low
> impact other than fixing up a few issues and giving implementations
> something they can point at for what the standard behavior is.
>
> So congratulations to everyone working on this :)
>
> I’ll get this into the PEPs repo and get it pushed.
>
>
> > On Nov 16, 2015, at 3:46 PM, Robert Collins 
> wrote:
> >
> > :PEP: XX
> > :Title: Dependency specification for Python Software Packages
> > :Version: $Revision$
> > :Last-Modified: $Date$
> > :Author: Robert Collins 
> > :BDFL-Delegate: Donald Stufft 
> > :Discussions-To: distutils-sig 
> > :Status: Draft
> > :Type: Standards Track
> > :Content-Type: text/x-rst
> > :Created: 11-Nov-2015
> > :Post-History: XX
> >
> >
> > Abstract
> > 
> >
> > This PEP specifies the language used to describe dependencies for
> packages.
> > It draws a border at the edge of describing a single dependency - the
> > different sorts of dependencies and when they should be installed is a
> higher
> > level problem. The intent is provide a building block for higher layer
> > specifications.
> >
> > The job of a dependency is to enable tools like pip [#pip]_ to find the
> right
> > package to install. Sometimes this is very loose - just specifying a
> name, and
> > sometimes very specific - referring to a specific file to install.
> Sometimes
> > dependencies are only relevant in one platform, or only some versions are
> > acceptable, so the language permits describing all these cases.
> >
> > The language defined is a compact line based format which is already in
> > widespread use in pip requirements files, though we do not specify the
> command
> > line option handling that those files permit. There is one caveat - the
> > URL reference form, specified in PEP-440 [#pep440]_ is not actually
> > implemented in pip, but since PEP-440 is accepted, we use that format
> rather
> > than pip's current native format.
> >
> > Motivation
> > ==
> >
> > Any specification in the Python packaging ecosystem that needs to consume
> > lists of dependencies needs to build on an approved PEP for such, but
> > PEP-426 [#pep426]_ is mostly aspirational - and there are already
> existing
> > implementations of the dependency specification which we can instead
> adopt.
> > The existing implementations are battle proven and user friendly, so
> adopting
> > them is arguably much better than approving an aspirational, unconsumed,
> format.
> >
> > Specification
> > =
> >
> > Examples
> > 
> >
> > All features of the language shown with a name based lookup::
> >
> >requests [security,tests] >= 2.8.1, == 2.8.* ; python_version <
> "2.7.10"
> >
> > A minimal URL based lookup::
> >
> >pip @
> https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
> >
> > Concepts
> > 
> >
> > A dependency specification always specifies a distribution name. It may
> > include extras, which expand the dependencies of the named distribution
> to
> > enable optional features. The version installed can be controlled using
> > version limits, or giving the URL to a specific artifact to install.
> Finally
> > the dependency can be made conditional using environment markers.
> >
> > Grammar
> > ---
> >
> > We first cover the grammar briefly and then drill into the semantics of
> each
> > section later.
> >
> > A distribution specification is written in ASCII text. We use a parsley
> > [#parsley]_ grammar to provide a precise grammar. It is expected that the
> > specification will be embedded into a larger system which offers framing
> such
> > as comments, multiple line support via continuations, or other such
> features.
> >
> > The full grammar including annotations to build a useful parse tree is
> > included at the end of the PEP.
> >
> > Versions may be specified according to the PEP-440 [#pep440]_ rules.
> (Note:
> > URI is defined in std-66 [#std66]_::
> >
> >version_cmp   = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' |
> '==='
> >version   = wsp* ( letterOrDigit | '-' | '_' | '.' | '*' )+
> >version_one   = version_cmp version wsp*
> >version_many  = version_one (wsp* ',' version_one)*
> >versionspec   = ( '(' version_many ')' ) | version_many
> >urlspec   = '@' wsp* 
> >
> > Environment markers allow making a specification only take effect in some
> > environments::
> >
> >marker_op = version_cmp | 'in' | 'not' wsp+ 'in'
> >python_str_c  = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' |
> > '-' | '_' | '*')
> >dquote= '"'
> >squote= 

Re: [Distutils] New Design Landed in Warehouse

[Marcus Smith]

what's the sharp thing hanging from the "P"
a device that measures the packaging cubes?  loads them?
:  )

On Fri, Nov 20, 2015 at 8:22 AM, Donald Stufft  wrote:

> As many of you may know, I’ve been working on Warehouse which is designed
> to replace the PyPI code base with something modern and maintainable. That
> is progressing but we’ve hit a particular milestone that I’m really excited
> by. With the help of the awesome nlhkabu [1][2] we’ve gotten a (still WIP)
> design that has now landed in the repository and has been deployed to
> warehouse.python.org and warehouse-staging.python.org. Check it out and
> see what you think. If you find any issues feel free to open them up on
> https://github.com/pypa/warehouse.
>
>
> [1] https://twitter.com/nlhkabu/
> [2] https://github.com/nlhkabu/
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Marcus Smith]

>
>
> Its included in the complete grammar, otherwise it can't be tested.
> Note that that the PEP body refers to the IETF document for the
> definition of URIs. e.g. exactly what you suggest.
>

doesn't this imply any possible URI can theoretically be a PEP440 direct
reference URI ?

Is that true?

It's unclear to me what PEP440's definition really is with words like "The
exact URLs and targets supported will be tool dependent"

Will "direct references" ever be well-defined? or open to whatever any tool
decides can be an artifact reference?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Marcus Smith]

as it is, this PEP defers the concept of a "Direct Reference URL" to PEP440.

but then PEP440 partially defers to PEP426's "source_url" concept, when it
says  "a direct URL reference may be a valid source_url entry"

do we expect PEP440 to be updated to fully own what a "Direct Reference
URL" can be?, since referring to PEP426 is now a dead-end path (and
partially replaced by this PEP)

On Mon, Nov 16, 2015 at 12:46 PM, Robert Collins 
wrote:

> :PEP: XX
> :Title: Dependency specification for Python Software Packages
> :Version: $Revision$
> :Last-Modified: $Date$
> :Author: Robert Collins 
> :BDFL-Delegate: Donald Stufft 
> :Discussions-To: distutils-sig 
> :Status: Draft
> :Type: Standards Track
> :Content-Type: text/x-rst
> :Created: 11-Nov-2015
> :Post-History: XX
>
>
> Abstract
> 
>
> This PEP specifies the language used to describe dependencies for packages.
> It draws a border at the edge of describing a single dependency - the
> different sorts of dependencies and when they should be installed is a
> higher
> level problem. The intent is provide a building block for higher layer
> specifications.
>
> The job of a dependency is to enable tools like pip [#pip]_ to find the
> right
> package to install. Sometimes this is very loose - just specifying a name,
> and
> sometimes very specific - referring to a specific file to install.
> Sometimes
> dependencies are only relevant in one platform, or only some versions are
> acceptable, so the language permits describing all these cases.
>
> The language defined is a compact line based format which is already in
> widespread use in pip requirements files, though we do not specify the
> command
> line option handling that those files permit. There is one caveat - the
> URL reference form, specified in PEP-440 [#pep440]_ is not actually
> implemented in pip, but since PEP-440 is accepted, we use that format
> rather
> than pip's current native format.
>
> Motivation
> ==
>
> Any specification in the Python packaging ecosystem that needs to consume
> lists of dependencies needs to build on an approved PEP for such, but
> PEP-426 [#pep426]_ is mostly aspirational - and there are already existing
> implementations of the dependency specification which we can instead adopt.
> The existing implementations are battle proven and user friendly, so
> adopting
> them is arguably much better than approving an aspirational, unconsumed,
> format.
>
> Specification
> =
>
> Examples
> 
>
> All features of the language shown with a name based lookup::
>
> requests [security,tests] >= 2.8.1, == 2.8.* ; python_version <
> "2.7.10"
>
> A minimal URL based lookup::
>
> pip @
> https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
>
> Concepts
> 
>
> A dependency specification always specifies a distribution name. It may
> include extras, which expand the dependencies of the named distribution to
> enable optional features. The version installed can be controlled using
> version limits, or giving the URL to a specific artifact to install.
> Finally
> the dependency can be made conditional using environment markers.
>
> Grammar
> ---
>
> We first cover the grammar briefly and then drill into the semantics of
> each
> section later.
>
> A distribution specification is written in ASCII text. We use a parsley
> [#parsley]_ grammar to provide a precise grammar. It is expected that the
> specification will be embedded into a larger system which offers framing
> such
> as comments, multiple line support via continuations, or other such
> features.
>
> The full grammar including annotations to build a useful parse tree is
> included at the end of the PEP.
>
> Versions may be specified according to the PEP-440 [#pep440]_ rules. (Note:
> URI is defined in std-66 [#std66]_::
>
> version_cmp   = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' |
> '==='
> version   = wsp* ( letterOrDigit | '-' | '_' | '.' | '*' )+
> version_one   = version_cmp version wsp*
> version_many  = version_one (wsp* ',' version_one)*
> versionspec   = ( '(' version_many ')' ) | version_many
> urlspec   = '@' wsp* 
>
> Environment markers allow making a specification only take effect in some
> environments::
>
> marker_op = version_cmp | 'in' | 'not' wsp+ 'in'
> python_str_c  = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' |
>  '-' | '_' | '*')
> dquote= '"'
> squote= '\\''
> python_str= (squote (python_str_c | dquote)* squote |
>  dquote (python_str_c | squote)* dquote)
> env_var   = ('python_version' | 'python_full_version' |
>  'os_name' | 'sys_platform' | 'platform_release' |
>  'platform_system' | 'platform_version' |
>  'platform_machine' | 

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Marcus Smith]

>
>
> > Will "direct references" ever be well-defined? or open to whatever any
> tool
> > decides can be an artifact reference?
>
> We can define the syntax without capturing all the tool support, which
> is what PEP-440 and thus this PEP does.
>

so, to be clear, what syntax for the URI portion does it define or require?
 (beyond it just being a valid URI)

it sounds like you're saying nothing?  i.e. although PEP440 says things
like it "may" be a sdist or a wheel target or a "source_url", its wide open
to whatever a tool may decide is a unique artifact reference?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Marcus Smith]

On Wed, Nov 18, 2015 at 11:42 AM, Donald Stufft <don...@stufft.io> wrote:

>
> On Nov 18, 2015, at 2:40 PM, Marcus Smith <qwc...@gmail.com> wrote:
>
>
>> > Will "direct references" ever be well-defined? or open to whatever any
>> tool
>> > decides can be an artifact reference?
>>
>> We can define the syntax without capturing all the tool support, which
>> is what PEP-440 and thus this PEP does.
>>
>
> so, to be clear, what syntax for the URI portion does it define or
> require?  (beyond it just being a valid URI)
>
> it sounds like you're saying nothing?  i.e. although PEP440 says things
> like it "may" be a sdist or a wheel target or a "source_url", its wide open
> to whatever a tool may decide is a unique artifact reference?
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> Only half way thinking about this right this moment, but I think so yes.
> It’s largely designed for private use cases which is why it’s not allowed
> on PyPI. It’s essentially a replacement for dependency_links.
>

practically speaking, isn't it also a future replacement for
"#egg=name" syntax in pip vcs urls?... i.e.  using  "name@"
instead?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PyPUG Tutorials

[Marcus Smith]

yea, I was thinking the same.
but we'll see how the initial reorg goes.

On Sun, Nov 15, 2015 at 9:32 PM, Nick Coghlan <ncogh...@gmail.com> wrote:

> On 16 November 2015 at 03:49, Marcus Smith <qwc...@gmail.com> wrote:
> >>
> >> To have the most success, the writers will certainly need feedback from
> >> subject matter experts, so the process will include 2 stages where we
> >> specifically ask for feedback from PyPA-Dev and Distutils-Sig:  1) To
> >> validate the initial proposal that covers the scope of the changes, and
> 2)
> >> to review the actual PRs to PyPUG for accuracy, when it's time for
> merging.
> >> I'll post again with more details as those stages occur.
> >
> > So, I'm back to post the initial proposal as mentioned above.
>
> That generally looks good to me, but I think we're going to need to
> keep the "Advanced topics" section in one form or another. Longer
> term, it might be possible to split them out into themed subsections
> (as the outline does for pip/easy_install and wheel/egg by moving them
> into the history section), but I don't think reorganising them is at
> all urgent, so that can be tackled after this initial rearrangement is
> done.
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PyPUG Tutorials

[Marcus Smith]

>
>
> To have the most success, the writers will certainly need feedback from
> subject matter experts, so the process will include 2 stages where we
> specifically ask for feedback from PyPA-Dev and Distutils-Sig:  1) To
> validate the initial proposal that covers the scope of the changes, and 2)
> to review the actual PRs to PyPUG for accuracy, when it's time for merging.
>   I'll post again with more details as those stages occur.
>


So, I'm back to post the initial proposal as mentioned above.

The proposal was put together by Daniel Beck (http://www.danieldbeck.com),
one of the writers Nicole (from the Warehouse team) was able to rally with
her call for volunteers [1].

It includes a *tentative* new outline, but keep in mind that however it
turns out, we will maintain redirects for the current critical links, like
the tool recommendations page, and the two current tutorials.

As work proceeds, it will take place as PRs to the develop branch of the
PyPUG (https://github.com/pypa/python-packaging-user-guide/), so people can
follow along and help in the technical review process.When the final
product is deemed worthy and ready, there will be a final call for review
before the merge of develop back into master.

Both Daniel and Nicole are now subscribed to distutils-sig, and can respond
to any questions, concerns, or feedback.

Here's the proposal, as quoted from an email from Daniel:

-

Scope
-
The guide is meant to answer this question: "How do Python, packages, and
packaging tools fit together?" The guide's not to meant to replace tools'
docs; rather, it's to show when and how these tools and other artifacts
work as an ecosystem.

Audience
-
I get the impression that these are the major audience groups we want to
help with the guide:

- Python users who are at an 'advanced beginner' level (e.g., know how to
import from the standard library, but want to learn to install a
third-party package for the first time
- Python users with intermediate experience who want to make and distribute
a package for the first time
- Python users with more experience who need a refresher or want to
understand what the "one way to do it" is right now
- Python users with lots of experience who want to take on some advanced or
specialized packaging task

Outline
--
I sketched out a gross outline for the guide. This is not complete,
sometimes speculative, and certainly subject to change, but I wanted to
give you an overall picture of how I imagine the guide could look when
we've had some time to work on this.

Again, because the existing content is so good, I don't expect much, if
anything, will be deleted—so just because it's not listed here doesn't mean
anything. Rather, I expect that stuff will be easier to find. :-)

1. About this guide
2. Getting help
3. Quick references (i.e., five minute tutorials)
  a. Installing a package
  b. Making a package
  c. Publish a package on PyPI
4. Intro to Packaging*
  a. Who is this for?
  b. Prereqs (making sure Python, etc. are working)
  c. What's a package?
  d. Where do packages come from?
  e. Installing a package for the first time
  f. Using a virtualenv
  g. Making a package for the first time
  h. Publishing your first package
5. The Packaging Ecosystem (concept docs)
  a. Overview of tools
  b. Developing packages
  c. Distributing packages
  d. Indices and caches
6. Practices and patterns (task docs)
  a. Making packages
  b. Publishing packages
  c. Using packages
7. History and legacy of packaging
  a. that whole distutils->setuptools->distribute->setuptools saga
  b. history of PyPA
  c. eggs vs wheels
  d. easy_install vs pip
  e. cheeseshop vs warehouse
8. Contributing to this guide
  a. Style guide

* This is the only part that I think functions as a continuous unit meant
to be read from start to end. I'm imagining something that would work as,
for example, a lesson plan for a PyLadies workshop. For the rest of the
guide, I want to work with the expectation that every page is a landing
page.

Timeline

Here's my current perspective on how quickly we can move on these things:

- In the next few days: contacting volunteers to let them know that things
are happening soon (Nicole)
- In the 7 or 8 days: decide on workflow questions (Nicole and Marcus),
amend project roles (Nicole and Marcus), and ask for distutils-sig approval
(Marcus)
- In the next two weeks: start opening issues and inviting volunteers to
start writing
- By the end of November: most, if not all, existing content could be
reorganized into a new outline
- By mid-December: have publishable versions of the new tutorials
- By the end of January: potentially finished?


[1] https://github.com/pypa/warehouse/issues/729
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Marcus Smith]

> Because even if we go with the entry-point-style Python
> hooks, the build frontends like pip will still want to spawn a child
> to do the actual calls -- this is important for isolating pip from the
> build backend and the build backend from pip, it's important because
> the build backend needs to execute in a different environment than pip
> itself, etc.

[...]
> Concretely, the way I imagine this would work is that pip would set up
> the build environment, and then it would run
>
>   build-environment/bin/python path/to/pip-worker-script.py 
>

fwiw, such a worker is what I was describing in an earlier thread with
Robert last work

https://mail.python.org/pipermail/distutils-sig/2015-October/027443.html

although I wasn't arguing for it in that context, but rather just using it
to be clear that a python api approach could still be used with build
environment isolation
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPA Roadmap

[Marcus Smith]

btw, I'm very aware that recent discussions may be changing the roadmap...
  :  )
I'm holding fast for the smoke to clear...


On Wed, Nov 4, 2015 at 4:42 PM, Marcus Smith <qwc...@gmail.com> wrote:

> FYI, I went ahead and merged it.
>
> https://www.pypa.io/en/latest/roadmap/
>
> Again, help appreciated from anyone to keep it accurate as things change
> (and they surely will)
>
> --Marcus
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed language for how build environments work in the new build system interface

[Marcus Smith]

this reads ok to me...

On Sun, Nov 8, 2015 at 9:20 PM, Nathaniel Smith  wrote:

> Hi all,
>
> Following the strategy of trying to break out the different
> controversial parts of the new build system interface, here's some
> proposed text defining the environment that a build frontend like pip
> provides to a project-specific build backend.
>
> Robert's PEP currently disclaims all of this as out-of-scope, but I
> think it's good to get something down, since in practice we'll have to
> figure something out before any implementations can exist. And I think
> the text below pretty much hits the right points.
>
> What might be controversial about this nonetheless is that I'm not
> sure that pip *can* reasonably implement all the requirements as
> written without adding a dependency on virtualenv (at least for older
> pythons -- obviously this is no big deal for new pythons since venv is
> now part of the stdlib). I think the requirements are correct, so...
> Donald, what do you think?
>
> -n
>
> 
>
> The build environment
> -
>
> One of the responsibilities of a build frontend is to set up the
> environment in which the build backend will run.
>
> We do not require that any particular "virtual environment" mechanism
> be used; a build frontend might use virtualenv, or venv, or no special
> mechanism at all. But whatever mechanism is used MUST meet the
> following criteria:
>
> - All requirements specified by the project's build-requirements must
> be available for import from Python.
>
> - This must remain true even for new Python subprocesses spawned by
> the build environment, e.g. code like::
>
> import sys, subprocess
> subprocess.check_call([sys.executable, ...])
>
>   must spawn a Python process which has access to all the project's
> build-requirements. This is necessary e.g. for build backends that
> want to run legacy ``setup.py`` scripts in a subprocess.
>
>   [TBD: the exact wording here will probably need some tweaking
> depending on whether we end up using an entrypoint-like mechanism for
> specifying build backend hooks (in which case we can assume that hooks
> automatically have access to sys.executable), or a subprocess-based
> mechanism (in which case we'll need some other way to communicate the
> path to the python interpreter to the build backend, e.g. a PYTHON=
> envvar). But the basic requirement is pretty much the same either
> way.]
>
> - All command-line scripts provided by the build-required packages
> must be present in the build environment's PATH. For example, if a
> project declares a build-requirement on `flit
> `_, then the following must
> work as a mechanism for running the flit command-line tool::
>
> import subprocess
> subprocess.check_call(["flit", ...])
>
> A build backend MUST be prepared to function in any environment which
> meets the above criteria. In particular, it MUST NOT assume that it
> has access to any packages except those that are present in the
> stdlib, or that are explicitly declared as build-requirements.
>
>
> Recommendations for build frontends (non-normative)
> ...
>
> A build frontend MAY use any mechanism for setting up a build
> environment that meets the above criteria. For example, simply
> installing all build-requirements into the global environment would be
> sufficient to build any compliant package -- but this would be
> sub-optimal for a number of reasons. This section contains
> non-normative advice to frontend implementors.
>
> A build frontend SHOULD, by default, create an isolated environment
> for each build, containing only the standard library and any
> explicitly requested build-dependencies. This has two benefits:
>
> - It allows for a single installation run to build multiple packages
> that have contradictory build-requirements. E.g. if package1
> build-requires pbr==1.8.1, and package2 build-requires pbr==1.7.2,
> then these cannot both be installed simultaneously into the global
> environment -- which is a problem when the user requests ``pip install
> package1 package2``. Or if the user already has pbr==1.8.1 installed
> in their global environment, and a package build-requires pbr==1.7.2,
> then downgrading the user's version would be rather rude.
>
> - It acts as a kind of public health measure to maximize the number of
> packages that actually do declare accurate build-dependencies. We can
> write all the strongly worded admonitions to package authors we want,
> but if build frontends don't enforce isolation by default, then we'll
> inevitably end up with lots of packages on PyPI that build fine on the
> original author's machine and nowhere else, which is a headache that
> no-one needs.
>
> However, there will also be situations where build-requirements are
> problematic in various ways. For example, a package author might
> accidentally leave off some crucial requirement despite 

Re: [Distutils] New PEP : dependency specification

[Marcus Smith]

>
>
> Maybe it would be clearer to drop the comment and newline handling
> stuff from the core requirement specifier syntax (declaring that
> newlines are simply a syntax error), and assume that there's some
> higher-level framing protocol taking care of that stuff?


that sounds right to me.



>   https://gist.github.com/njsmith/ed74851c0311e858f0f7
>
> (Nice to see Metadata-Version: 2.0 getting some real-world use! I
> guess???


the use of 2.0 already is odd
see here for old discussion in the wheel tracker:
https://bitbucket.org/pypa/wheel/issues/96/metadata-version-20
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Marcus Smith]

>
>
>  PEP-345 doesn't
> describe Provides-Extra, which pkg_resources uses when parsing
> .dist-info directories as well


fwiw, this provides a bit of history on the "Provides-Extra":
https://github.com/pypa/interoperability-peps/issues/44
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Marcus Smith]

>
>
> I'm not sure this is the syntax that I would have come up with, but I
> guess it's too late to put the genie back in the bottle, so this PEP
> should have some way to cope with these things?


why would this PEP deal with this?
the higher level PEP that builds on top of this would bump the wheel
metadata version (probably to 3.0, due to the jump to 2.0 already)
*then* tools will cope with this based on the metadata version
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Marcus Smith]

>
>
> So both the abstract build system PEP and donalds setup.py interface
> depend on having a bootstrap dependency list written into a file in
> the source tree.


your build PEP said stuff like this  "Additional data *may* be included,
but the ``build_requires`` and ``metadata_version`` keys must be present"

that leads me to think you need more than just this specification of a
single dependency.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Marcus Smith]

>
>
>
> No - it specifies the serialisation format for
> names/specifiers/extras/markers that is in common use, but doesn't
> specify a programming API. It is intended as an interop building
> block,


I'm not not talking about programming API.
this PEP would set the format used in interop formats, as you say, but also
doesn't it effectively standardize the format used in the pip UI?
or maybe it's just that pip could easily say..  "our syntax for
dependencies mirrors PEPXX exactly"
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Marcus Smith]

>
>
> The language defined is a compact line based format which is already in
> widespread use


this is the most critical thing for me, and the reason this approach seems
more attractive than the path of PEP426, although I'd certainly like to see
Nick's reaction.

PEP426 tries to cover how names/specifiers/extras/markers would be put
together in abstract "in-memory representation" (that can be serialized to
json), but it's left open to pip (and other tools) to lay down a standard
(via implementation) for how these pieces are put together and used by
users.

this PEP would dictate both, right?  the user way, and the internal
metadata way
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Why github and bitbucket?

[Marcus Smith]

>
>
>
> Basically: Historical reasons. The name “PyPA” was a joke by the
> pip/virtualenv developers and it was only pip and virtualenv so it was on
> Github.


here's an anecdote per the pypa.io history page,  'Other proposed names
were “ianb-ng”, “cabal”, “pack” and “Ministry of Installation”'

https://www.pypa.io/en/latest/history/

maybe even funnier that we have a history page, but it's easy to forget all
that's happened, so I made one awhile back...
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New PEP : dependency specification

[Marcus Smith]

sorry, I feel like I have confirm my translation of your intro paragraph  :
 )
maybe it will help some others...


ended up with a hard dependency on this


my understanding is that you were depending on having PEP426 metadata, e.g.
for build_requires.
since this PEP, as you say doesn't handle the "higher level problem" of
specifying the types of dependencies (like PEP426 does), I guess you'll
have another PEP in the works as well on top of this?  and then your build
PEP would be depending on that.


bootstrap thing


you mean your other PEP idea for supporting any build system using the
indirect mapping thing to processes...



> 'smaller step' or not


you mean Donald's idea of using "setup.py" as the build interface for now


Donald has graciously agreed to be a BDFL-delegate for it.
>

doesn't Nick actually don this delegation?... not that Donald wouldn't be
great.



> and there are already existing
> implementations of the dependency specification which we can instead adopt.
> The existing implementations are battle proven and user friendly


for reference, which ones are those?


> The language defined is a compact line based format which is already in
> widespread use in pip requirements files

to be clear though, this PEP doesn't commit to how lines are put together
in the metadata.   theoretically, this spec could be consistent with a
higher-level spec that used json, right?  probably better to refer to "pip
syntax" than "pip requirements files"
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The Update Framework, integrate into PyPI?

[Marcus Smith]

>
> answering basic questions can take time away from making important
> improvements?
>

to be fair, distutils-sig is mentioned as a user support list on the
"Python Packaging User Guide"
a few years back, there was a debate on splitting it between a user and
planning list, but no traction there.
one concern was that the user list wouldn't have enough experts
participating to answer the questions.

--Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPA Roadmap

[Marcus Smith]

FYI, I went ahead and merged it.

https://www.pypa.io/en/latest/roadmap/

Again, help appreciated from anyone to keep it accurate as things change
(and they surely will)

--Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPA Roadmap

[Marcus Smith]

>
>
>
> Shouldn't Warehouse be mentioned there?
>

Indeed.  I'll add it.
thanks
Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPA Roadmap

[Marcus Smith]

>
>
> One question - would it be worth having a "design principles" section
> for notes about things like how we want to allow Warehouse/PyPI to
> publish metadata from the distribution files, so distribution formats
> should include static metadata for those values?


maybe.  we can certainly beef up the summaries and status comments.

I'm wary that trying to maintain "design principles" is too much for this
document (at least more than I'd want to maintain).

btw, I just added a sentence to the sdist section about the static metadata
discussion.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PyPA Roadmap

[Marcus Smith]

Based on discussions in another thread [1], I've posted a PR to pypa.io for
a "PyPA Roadmap"

PR: https://github.com/pypa/pypa.io/pull/7
built version:  http://pypaio.readthedocs.org/en/roadmap/roadmap/

To be clear, I'm not trying to dictate anything here, but rather just
trying to mirror what I think is going on for the sake of new (or old)
people, who don't have a full picture of the major todo items.

I'm asking for help to make this as accurate as possible and to keep it
accurate as our plans change.

thanks,
Marcus

[1]
https://mail.python.org/pipermail/distutils-sig/2015-October/027346.html ,
 although it seems a number of emails in this thread never made it to the
archive due to the python mail server failure.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] BDFL Delegates for distutils-sig PEPs

[Marcus Smith]

>
>
> =
> Whenever a new PEP is put forward on distutils-sig, any PyPA core
> reviewer that believes they are suitably experienced to make the final
> decision on that PEP may offer to serve as the BDFL's delegate (or
> "PEP czar") for that PEP. If their self-nomination is accepted by the
> other PyPA core reviewer, the lead PyPI maintainer and the lead
> CPython representative on distutils-sig, then they will have the
> authority to approve (or reject) that PEP.
> =
>

Nick:
just be clear, if nobody nominates themselves, then you still remain (by
default) the active delegate who's responsible for ruling on a Pypa-related
PEP?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP

[Marcus Smith]

>
>
> > 4) Although using a process interface is not necessarily a problem, I
> don't
> > agree with your point on why a python interface would be unworkable.
> You're
> > assuming that pip would try to import all the build tools (for every
> > dependency it's processing) in the master process.  An alternative could
> be
> > that pip would have it's own tool (that runs as a subprocess in an
> isolated
> > env) that knows how to load and work with python build interfaces.   You
> > could argue that a python api is  an advantage, because build tools
> aren't
>
> That would mean that pip has to have the same exact version of it embedded
> in the environment that build tools will be run in, which it doesn't have
> today.
>

sorry, I can't make clear sense of your sentence here.  :  )

I'll just explain my point again.

pip doesn't necessarily have to "interact with many different versions of
the same build tool during a single invocation" if for example it's
subprocessing the interactions to some "pip-build" tool that handles the
imports and use of the python API.  I.e. pips calls some "pip-build" too
(per build), which does the import, not pip itself.

and again, it's not about arguing for this idea, but just that your
"in-process APIs are harder" argument doesn't decide the matter.


>
> I see no problem with evolving them in lockstep,


it's unnecessarily complex IMO.  if they're really in lockstep, then
they're one thing it seems to me.

I'd rather avoid the chance for a bug where something tries to parse a
> v2 schema build
> description with a v1 schema parser.


but it won't happen?  the pypa.yaml schema version would determine the
parser version that's used.



> > 7) it's unclear when pip get's to run "dist-info" and when the result
> might
> > be different.   For example, we've discussed that run time dependencies
> may
> > get clarifed *after* the build process so this command might produce
> > different results at different times?
>
> Pip would run dist-info when determining the install-requires and
> extras for the package.


you're not addressing the point about how the act of building can create
new run time dependencies, per the whole long discussion with Nathaniel
 recently  (his draft deals with this matter explicitly)

--Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Second draft of a plan for a new source tree / sdist format

[Marcus Smith]

>
>
> - Will allow for both static and dynamic specification of build
> dependencies


I think you need to fill in the story on dynamic dependencies, or otherwise
this PEP will be a mystery to most people.

I *think* I understand your motivation for this, based on hearing your plan
(in another thread) of putting c libs into wheels, and actually declaring
them as dependencies... and because these libs can vary based on build
settings,  your dependencies are "dynamic"... but all that needs to be
explained in the PEP

--Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A smaller step towards de-specializing setuptools/distutils

[Marcus Smith]

>
>
> * This essentially doesn't solve any of the dynamic vs static metadata
> issues
>

Nathaniel's grand plan requires dynamic run-time dependencies, so to be
clear, this plan doesn't help the case that instigated most of the recent
discussion, right?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A smaller step towards de-specializing setuptools/distutils

[Marcus Smith]

>
> > Nathaniel's grand plan requires dynamic run-time dependencies, so to be
> > clear, this plan doesn't help the case that instigated most of the recent
> > discussion, right?
>
> It should still solve that problem because it allows a project to
> dynamically decide what dependencies they have (just like they can today)
>

help me out here... how can we dynamically construct dependencies as we're
building wheels today?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP

[Marcus Smith]

>
>
> pip doesn't necessarily have to "interact with many different versions of
> the same build tool during a single invocation" if for example it's
> subprocessing the interactions to some "pip-build" tool that handles the
> imports and use of the python API.  I.e. pips calls some "pip-build" too
> (per build), which does the import, not pip itself.
>

sorry, there's 2 confusing misspells in my sentence.  here it is again:

pip doesn't necessarily have to "interact with many different versions of
the same build tool during a single invocation" if for example it's
sub-processing the interactions to some "pip-build" tool that handles the
imports and use of the python API.  I.e. pip calls some "pip-build" tool
(per build), which does the import, not pip itself.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A smaller step towards de-specializing setuptools/distutils

[Marcus Smith]

>
>
> since a wheel is created by executing setup.py, you’d just have your build
> tool dynamically output different wheels based on the system you’re
> building on (or whatever axis is causing the dynamic dependencies).


understood, but I guess I was under the [mistaken?] impression, that
dynamic support is needed literally in the wheel build step.

for example, going with your idea above, that we need to support 3rd party
tools being dynamic in the "setup.py bdist_wheel" step,  not simply in the
"setup.py dist-info" step.

and as it is, pip does't understand this possibility.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A smaller step towards de-specializing setuptools/distutils

[Marcus Smith]

>
>
> > for example, going with your idea above, that we need to support 3rd
> party
> > tools being dynamic in the "setup.py bdist_wheel" step, not simply in the
> > "setup.py dist-info" step.
> >
> > and as it is, pip does't understand this possibility.
>
> So in the new path, we wouldn’t call ``setup.py dist-info`` for “normal”
> installations, we’d compile a wheel and inspect that for dependencies.
>

ok, so a pretty significant change to pip's resolution internals.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A smaller step towards de-specializing setuptools/distutils

[Marcus Smith]

>
>
> > so, Robert, to be clear, you think 3rd party build tools can get the
> > dynamic-dependency support they need just by implementing something
> dynamic
> > via "setup,py dist-info"? (and it doesn't need to happen literally in the
> > wheel build step?)
>
> Sure -- 'dist-info' would be run in the same environment as
> bdist_wheel, a few seconds before bdist_wheel was run (unless the
> output from dist-info convinces us that we need a different version of
> this package and so we skip running bdist_wheel entirely). So at least
> in principle, anything bdist_wheel can do/know/figure-out is something
> that dist-info could also do/know/figure-out. The question about
> whether to provide a 'dist-info' step is whether the optimization it
> provides is worth the extra complexity it imposes on build systems and
> the opportunities for what Donald called "race conditions" (=
> accidental or intentional skew between the output of dist-info and the
> actual metadata in the eventual wheel).
>

thankyou!! for the clarity  :  )
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A smaller step towards de-specializing setuptools/distutils

[Marcus Smith]

>
>
> However, the big step you're proposing that I think is fundamentally
> unsound is that of requiring a wheel be built before dependencies can
> be queried.


so, Robert, to be clear, you think 3rd party build tools can get the
dynamic-dependency support they need just by implementing something dynamic
via "setup,py dist-info"? (and it doesn't need to happen literally in the
wheel build step?)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Please don't impose additional barriers to participation (was: build system abstraction PEP)

[Marcus Smith]

my intention certainly wasn't to try to exclude anybody.  for me, it's the
practical matter of the PR UI being more effective than a mailing list
thread (in this case referring to a gist), and that we can track proposals
easier and link to them from issues (in that same repo) and other PyPA
docs.

also, to be clear, this isn't about thinking PyPA would sidestep the PEP
approval process all together.  It's about managing drafts, reviews and
updates into that process.

certainly, this isn't my decision alone to make this change... just stating
my preference, and hoping to get others to agree.   Part of my motivation
for bringing it up, was do due to trying write a "PyPA Roadmap" recently
for pypa.io, and wanting it to be easier to track and link to ideas that
people are coming up with (ideas that don't immediately start as draft PEPs)

--Marcus


On Tue, Oct 27, 2015 at 10:02 PM, Ben Finney <ben+pyt...@benfinney.id.au>
wrote:

> Marcus Smith <qwc...@gmail.com> writes:
>
> > 1) *Please*, *please*, *please* let's start doing PEP conversations as
> > PRs to pypa/interoperability-peps : )
>
> Please keep the conversation on a mailing list where one can participate
> without needing to sign up with any particular service provider.
>
> Your proposal would have the effect of excluding people from the
> conversation if they don't agree to have a GitHub account. I think it's
> valuable to avoid that barrier to entry, needing only an email account.
>
> --
>  \   “'Tis strange, — but true; for truth is always strange; / |
>   `\Stranger than fiction.” —“Lord” George Gordon Noel Byron, _Don |
> _o__)Juan_ |
> Ben Finney
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP

[Marcus Smith]

>
>
> Current draft text in rendered form at:
> https://gist.github.com/rbtcollins/666c12aec869237f7cf7
>
>
Thanks for working on this.
Overall I like the idea, but have some comments/questions


1)  *Please*, *please*, *please* let's start doing PEP conversations as PRs
to pypa/interoperability-peps :  )  There's a place in there for unnumbered
PEPs.  Review will be better and faster IMO as PRs.  If the PR gets too
cluttered with old conversations, just start clean with a new PR.   We can
announce the PRs here, but don't discuss them on the mailing list.   The
same goes for trimming down PEP426 as you mention.   Let's do that as a PR
to pypa/interoperability-peps to the existing document.

2) Ditto on Ralf's concerns about readability.  I only really understood it
after seeing the examples you gave to Ralf (they need to be in the PEP, not
just in response to Ralf).   There's a few places I'd want to alter the
wording, but I likely won't bother, unless it's done as a PR.

3) You refer to "source distribution".  What does that mean exactly in this
PEP?  just the current setuptools notion?

4) Although using a process interface is not necessarily a problem, I don't
agree with your point on why a python interface would be unworkable.
You're assuming that pip would try to import all the build tools (for every
dependency it's processing) in the master process.  An alternative could be
that pip would have it's own tool (that runs as a subprocess in an isolated
env) that knows how to load and work with python build interfaces.   You
could argue that a python api is  an advantage, because build tools aren't
forced to grow a certain cli for pip, they just have to add a shim module
that conforms to the interface.   But my point is not to argue for the
Python API, but for you to remove an argument that from what I can tell,
isn't really an argument for one or the other.

5) at one pt you said "--dump-build-requires would just output a constant
string: {"schema_version": "2.0", "build_requires": []}"   you meant
"metadata_version", right?

6)  Can you explain better why we need to manage a pypa.yaml schema version
*and* a build description schema version.   Why not just a version for
pypa.yaml, and if anything changes (in the yaml schema or the build
"description" api),  then just bump the version.

7) it's unclear when pip get's to run "dist-info" and when the result might
be different.   For example, we've discussed that run time dependencies may
get clarifed *after* the build process so this command might produce
different results at different times?

--Marcus



> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] New PyPUG Tutorials

[Marcus Smith]

As some of you may already know, Nicole from the Warehouse team has
initiated an effort to improve our packaging tutorials, and have them
linked in the new Warehouse UI (https://github.com/pypa/warehouse/issues/729
).

Although originally the idea was to create additional tutorials (but still
maintain consistency with the "Python Packaging User Guide", aka "PyPUG"),
after further discussions, we've decided to just improve the 2 existing
PyPUG sections that we have (for installation and distribution), with the
goal being to improve the accessibilty for new users, and in general to
improve the quality and overall style and experience.

Nicole has already gathered a group of volunteer writers, and there's a
plan in place to manage the process from planning to release.

To have the most success, the writers will certainly need feedback from
subject matter experts, so the process will include 2 stages where we
specifically ask for feedback from PyPA-Dev and Distutils-Sig:  1) To
validate the initial proposal that covers the scope of the changes, and 2)
to review the actual PRs to PyPUG for accuracy, when it's time for merging.
  I'll post again with more details as those stages occur.

As someone who has slowly chipped away on the PyPUG for the last 2 years,
I'm excited about the prospect of being able to make a big jump in quality
very quickly.

--Marcus

P.S. for anyone here who wants to be on the core writing team, see
https://github.com/pypa/warehouse/issues/729
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Time for a setuptools_lite??

[Marcus Smith]

>
>
>> Would it makes sense to start a roadmap doc/repo under the PyPA account
> so the current grand vision is written down in a very high-level overview
>

I think it makes sense.

we had such a page for awhile at https://www.pypa.io
I recently dropped it because nobody was really maintaining it and it was
pretty stale.
we still have a high-level set of goals there, but the section that had
literal todos and links to PRs etc. is gone.

I'm all for adding it back, but we need all the major players to commit to
maintaining it...
and everyone needs to be somewhat noisy about making sure we keep it up to
date this time.

Mainly, I I think it should consist of a set of links to Issues and PRs in
the PyPA PEP Repor [1], and links to major things going on in pip,
setuptools, etc...

Give me some +1s folks, and I'm willing to crank it up again... and rally
for help to make it as accurate as I know to make it.

[1] https://github.com/pypa/interoperability-peps
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Meeting info re: sdists

[Marcus Smith]

thanks for the summary!


  * Things that have reason to change (deps) are more reasonable to be
> dynamic (even with PEP-426 markers there are exceptions)
>

as we know, for *many* cases, run-time deps aren't dynamic.
is there a consensus for those cases?  exist in the sdist metadata? or no?
or maybe with the "auto-building" Donald mentioned,  we'd rely solely on
wheel "built" metadata as the vehicle for dependency resolution

also, did Nathaniel's idea of uploading c libraries into PyPI as python
wrapper projects get discussed?

--Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What metadata does pip actually need about sdists?

[Marcus Smith]

> 2) after unpacking this sdist it then calls 'setup.py egg_info' to get
> the full metadata for the wheel


I wouldn't say "get the full metadata for the wheel".  it's not a wheel
yet.
`egg_info` run so we can use the pkg_resources api to find the dependencies.



> Specifically what it does with this is
> extract the setup_requires and install_requires fields


specifically, we call `requires` on pkg_resources distribution objects
https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L583



> 3) eventually it actually builds the package, and this produces a
> wheel (or wheel equivalent) that has its own metadata (which often
> matches the metadata from egg_info in step (2), but not always)
>

"not always"?  not following your point
they're 2 different formats, but should contain the same essential
information.
here's the wheel code that does the conversion
https://bitbucket.org/pypa/wheel/src/1cb7374c9ea4d5c82992f1d9b24cf7168ca87707/wheel/metadata.py?at=default=file-view-default#metadata.py-90



> name and version before it runs egg_info, I assume that what this
> means is that it's crucial for pip to have static access to dependency
> information?


yes



> It would be somewhat convenient if sdists did list their binary
> dependencies:


not sure about your insertion of "binary" here.
pip is concerned with finding python project dependencies (i.e. name and
version constraints) in the sdist
and then based on the current install environment, it will further
constrain the wheels chosen based on architecture and python implementation.
and to be perfectly clear, none of this deals with non-python OS/Distro
requirements.



> 3) if any of the packages-to-be-installed are sdists, then fetch them,
> run egg_info or build them or whatever to get their real dependencies,
> add these to the graph, and go to step 1
>

this is the pain we don't want in the future.

since we're simply talking about name/version constraints (not
platform/python), It's hard to conceive that we'd agree on an sdist spec
that didn't include that.



> Do the relevant pip maintainers
> even read this mailing list? :-)
>

I try  :  )
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What metadata does pip actually need about sdists?

[Marcus Smith]

> since we're simply talking about name/version constraints (not
platform/python)

let me clarify that... since I imagine someone might bring up markers in
response...
markers allow you to vary your name/version dependencies by environment
(for example, by platform and python version),
but to be clear, it's still not a "binary" dependency declaration... i.e.
the dependency is still declared by name and a version specifier alone.


On Sun, Oct 11, 2015 at 10:48 AM, Marcus Smith <qwc...@gmail.com> wrote:

>
>
>> 2) after unpacking this sdist it then calls 'setup.py egg_info' to get
>> the full metadata for the wheel
>
>
> I wouldn't say "get the full metadata for the wheel".  it's not a wheel
> yet.
> `egg_info` run so we can use the pkg_resources api to find the
> dependencies.
>
>
>
>> Specifically what it does with this is
>> extract the setup_requires and install_requires fields
>
>
> specifically, we call `requires` on pkg_resources distribution objects
> https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L583
>
>
>
>> 3) eventually it actually builds the package, and this produces a
>> wheel (or wheel equivalent) that has its own metadata (which often
>> matches the metadata from egg_info in step (2), but not always)
>>
>
> "not always"?  not following your point
> they're 2 different formats, but should contain the same essential
> information.
> here's the wheel code that does the conversion
>
> https://bitbucket.org/pypa/wheel/src/1cb7374c9ea4d5c82992f1d9b24cf7168ca87707/wheel/metadata.py?at=default=file-view-default#metadata.py-90
>
>
>
>> name and version before it runs egg_info, I assume that what this
>> means is that it's crucial for pip to have static access to dependency
>> information?
>
>
> yes
>
>
>
>> It would be somewhat convenient if sdists did list their binary
>> dependencies:
>
>
> not sure about your insertion of "binary" here.
> pip is concerned with finding python project dependencies (i.e. name and
> version constraints) in the sdist
> and then based on the current install environment, it will further
> constrain the wheels chosen based on architecture and python implementation.
> and to be perfectly clear, none of this deals with non-python OS/Distro
> requirements.
>
>
>
>> 3) if any of the packages-to-be-installed are sdists, then fetch them,
>> run egg_info or build them or whatever to get their real dependencies,
>> add these to the graph, and go to step 1
>>
>
> this is the pain we don't want in the future.
>
> since we're simply talking about name/version constraints (not
> platform/python), It's hard to conceive that we'd agree on an sdist spec
> that didn't include that.
>
>
>
>> Do the relevant pip maintainers
>> even read this mailing list? :-)
>>
>
> I try  :  )
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What metadata does pip actually need about sdists?

[Marcus Smith]

>
>
> So instead, the current plan is that we're going
> to drop the libraries inside a wheel and upload it to PyPI:
>

aha... ok, now it's clearer where you're coming from.
but using what platform in the wheel tag?
linux wheels are blocked currently

regardless of the tagging though, I'm not so sure about using PyPI like
this to hold all these "python" wrapper packages for binary extensions
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What metadata does pip actually need about sdists?

[Marcus Smith]

>
>
> it's a fact of life that the same
> source release may be configured in different ways that create
> different resulting dependencies. NumPy is one example of this, but
> it's hardly unusual


I've tried to respond to this point twice.. but let me try again. :  )
If I'm confused, please help me.

aren't the "different resulting dependencies" non-python, non-pypi system
dependencies?

the dependencies the sdist would cover are the python/pypi dependencies,
and they don't usually vary based on building extensions

as for representing non-python distro dependencies and build variances, I
mentioned before that PEP426 extensions might be used to cover that, but
that's still an open question I think..

--Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Marcus Smith]

>
> But the different builds for the different configurations end up with
> different metadata. If I'm understanding right, the whole point of
> "source wheels" is that they have all the static metadata that pip
> needs in order to make decisions, and this has to match the resulting
> wheels -- right?
>

I think we're largely talking about variances in "external" non-python
system dependencies (and their build settings).

PEP426 currently doesn't cover this in the core metadata, so as it stands,
any 2.0 sdist couldn't exhaust these build variances in it's core metadata.

There has been some discussion on how to represent external dependencies.
In brief, I think the going idea is that it would be through extensions (
https://www.python.org/dev/peps/pep-0426/#metadata-extensions), not in the
core python metadata, and the various groups (distro folks, science folks,
etc..) would implement these themselves to fulfill their needs...

Assuming they did implement such an extension, it would exist in the sdist,
and for cases like numpy likely support some notion of "build options", and
hence allow for a 1 to many mapping between sdist and binary wheels.

Marcus


> The way I'm imagining it is that there are multiple levels of metadata
> staticness:
>
> package name, author, description, ...
>   static in: VCS checkouts, source releases, source wheels, wheels
> package version
>   static in: source releases, source wheels, wheels
> package dependencies
>   static in: source wheels, wheels
> environment tag
>   static in: wheels
>
> > Of course, there *is* an unsolved issue here, which is how we manage
> > compatibility for wheels at the level needed for numpy. But I thought
> > the discussion on that was ongoing? I'm concerned that this proposal
> > is actually about bypassing that discussion, and instead trying to
> > treat incompatibly linked wheels as "different" in terms of project
> > metadata, which I think is the wrong way of handling things. I note
> > that Christoph Gohlke's numpy builds are tagged with a "+mkl" local
> > version modifier - that's presumably intended to mark the fact that
> > they are built with an incompatible runtime - but that's a misuse of
> > local versions (and I've found it causes niggling issues with how pip
> > recognises upgrades, etc).
>
> Yeah, that's not a good long term solution -- it needs to be moved
> into the metadata (probably by creating an MKL wheel and then making
> the numpy wheel depend on it). That's exactly the problem :-)
>
> > So, in summary: Your points above don't seem to me to in any way
> > preclude having a single numpy source wheel, and a number of (mutually
> > incompatible, but the same in terms of project and version) binary
> > wheels.
>
> Maybe I have misunderstood: does it actually help pip at all to have
> static access to name and version, but not to anything else? I've been
> assuming not, but I don't think anyone's pointed to any examples yet
> of the problems that pip is encountering due to the lack of static
> metadata -- would this actually be enough to solve them?
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Marcus Smith]

Can you clarify the relationship to PEP426 metadata?
There's no standard for metadata in here other than what's required to run
a build hook.
Does that imply you would have each build tool enforce their own convention
for where metadata is found?

On Thu, Oct 1, 2015 at 9:53 PM, Nathaniel Smith  wrote:

> Hi all,
>
> We realized that actually as far as we could tell, it wouldn't be that
> hard at this point to clean up how sdists work so that it would be
> possible to migrate away from distutils. So we wrote up a little draft
> proposal.
>
> The main question is, does this approach seem sound?
>
> -n
>
> ---
>
> PEP: ??
> Title: Standard interface for interacting with source trees
>and source distributions
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nathaniel J. Smith 
> Thomas Kluyver 
> Status: Draft
> Type: Standards-Track
> Content-Type: text/x-rst
> Created: 30-Sep-2015
> Post-History:
> Discussions-To: 
>
> Abstract
> 
>
> Distutils delenda est.
>
>
> Extended abstract
> =
>
> While ``distutils`` / ``setuptools`` have taken us a long way, they
> suffer from three serious problems: (a) they're missing important
> features like autoconfiguration and usable build-time dependency
> declaration, (b) extending them is quirky, complicated, and fragile,
> (c) you are forced to use them anyway, because they provide the
> standard interface for installing python packages expected by both
> users and installation tools like ``pip``.
>
> Previous efforts (e.g. distutils2 or setuptools itself) have attempted
> to solve problems (a) and/or (b). We propose to solve (c).
>
> The goal of this PEP is get distutils-sig out of the business of being
> a gatekeeper for Python build systems. If you want to use distutils,
> great; if you want to use something else, then the more the merrier.
> The difficulty of interfacing with distutils means that there aren't
> many such systems right now, but to give a sense of what we're
> thinking about see `flit `_ or
> `bento
> `_. Fortunately, wheels have now
> solved many of the hard problems here -- e.g. it's no longer necessary
> that a build system also know about every possible installation
> configuration -- so pretty much all we really need from a build system
> is that it have some way to spit out standard-compliant wheels.
>
> We therefore propose a new, relatively minimal interface for
> installation tools like ``pip`` to interact with package source trees
> and source distributions.
>
>
> Synopsis and rationale
> ==
>
> To limit the scope of our design, we adopt several principles.
>
> First, we distinguish between a *source tree* (e.g., a VCS checkout)
> and a *source distribution* (e.g., an official snapshot release like
> ``lxml-3.4.4.zip``).
>
> There isn't a whole lot that *source trees* can be assumed to have in
> common. About all you know is that they can -- via some more or less
> Rube-Goldbergian process -- produce one or more binary distributions.
> In particular, you *cannot* tell via simple static inspection:
> - What version number will be attached to the resulting packages (e.g.
> it might be determined programmatically by consulting VCS metadata --
> I have here a build of numpy version "1.11.0.dev0+4a9ad17")
> - What build- or run-time dependencies are required (e.g. these may
> depend on arbitrarily complex configuration settings that are
> determined via a mix of manual settings and auto-probing)
> - Or even how many distinct binary distributions will be produced
> (e.g. a source distribution may always produce wheel A, but only
> produce wheel B when built on Unix-like systems).
>
> Therefore, when dealing with source trees, our goal is just to provide
> a standard UX for the core operations that are commonly performed on
> other people's packages; anything fancier and more developer-centric
> we leave at the discretion of individual package developers. So our
> source trees just provide some simple hooks to let a tool like
> ``pip``:
>
> - query for build dependencies
> - run a build, producing wheels as output
> - set up the current source tree so that it can be placed on
> ``sys.path`` in "develop mode"
>
> and that's it. We teach users that the standard way to install a
> package from a VCS checkout is now ``pip install .`` instead of
> ``python setup.py install``. (This is already a good idea anyway --
> e.g., pip can do reliable uninstall / upgrades.)
>
> Next, we note that pretty much all the operations that you might want
> to perform on a *source distribution* are also operations that you
> might want to perform on a source tree, and via the same UX. The only
> thing you do with source distributions that you don't do with source
> trees is, well, distribute them. There's all kind of metadata you
> could imagine including in a 

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Marcus Smith]

>
>
>  So downstream distributors can download an sdist - or even a tarball of a
> VCS tag, if they're being strict about it - build wheels from that using
> the config in this proposal, and then transform the wheels into their own
> package format.
>

this has wheel itself being the interoperability standard.
the going idea is that there would be a metadata artifact ("pydist.json" in
PEP426), that provides the standard across different formats.


> Longer term, I think the answer is sdist 2.0 which has proper metadata
> inside of it (name, version, dependencies, etc) but which also includes a
> hook like this PEP has to specify the build system
>
> I hadn't heard of this before - is it something that's being worked on?
>

I haven't seen any PEP's for it yet.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Marcus Smith]

>
>> The first thing that immediately stood out to me, is that it's
>> recommending
>> that downstream redistributors like Debian, Fedora, etc utilize Wheels
>> instead
>> of the sdist to build their packages from. However, that is not really
>> going to
>> fly with most (all?) of the downstream redistributors. Debian for
>> instance has
>> policy that requires the use of building all of it's packages from
>> Source, not
>> from anything else and Wheels are not a source package. While it can
>> theoretically work for pure python packages, it quickly devolves into a
>> mess
>> when you factor in packages that have any C code what so ever.
>>
>
> So wouldn't they then download the sdist, build a wheel as an
> intermediate, and then generate the .deb file?
>

the new goal I think was to have standardized metadata immediately
available in an sdist, and get away from the model, that you had to run a
build step, before you had a metadata artifact.
so here,  you'd have to build a wheel (potentially one with binary
extensions) just to know what the metadata is? that doesn't sound right.



> I mean as long as people upload an sdist for those that want to build from
> source and a wheel for convenience -- which is probably what most people
> providing wheels do anyway -- then I don't see the problem.
>
>
>>
>> Overall, this feels more like a sidegrade than an upgrade. One major theme
>> throughout of the PEP is that we're going to push to rely heavily on
>> wheels as
>> the primary format of installation. While that works well for things like
>> Debian, I don't think it's going to work as wheel for us. If we were only
>> distributing pure python packages, then yes absolutely, however given
>> that we
>> are not, we have to worry about ABI issues. Given that there is so many
>> different environments that a particular package might be installed into,
>> all
>> with different ABIs we have to assume that installing from source is still
>> going to be a primary path for end users to install and that we are never
>> going
>> to have a world where we can assume a Wheel in a repository.
>>
>> One of the problems with the current system, is that we have no mechanism
>> by
>> which to determine dependencies of a source distribution without
>> downloading
>> the file and executing some potentially untrusted code. This makes
>> dependency
>> resolution harder and much much slower than if we could read that
>> information
>> statically from a source distribution. This PEP doesn't offer anything in
>> the
>> way of solving this problem.
>>
>
> Isn't that what the requirements and requirements-file fields in the
> _pypackage file provide? Only if you use that requirements-dynamic would it
> require execcuting arbitrary code to gather dependency information, or am I
> missing something?
>

those are just requirements to run the build hook, not run time
dependencies.



> Is there another proposal I'm unaware for the sdist -> wheel step that is
> build tool-agnostic?
>

PEP426 talks about it some
https://www.python.org/dev/peps/pep-0426/#metabuild-system
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Marcus Smith]

>
>
> If you're interested, I'm happy to directly collaborate on this PEP if
> it's in
> a github repository somewhere or something. There's an interoptability repo
>

btw, the repo he's talking about  is here:
https://github.com/pypa/interoperability-peps
it has a convention about where to add pep ideas that have no number yet
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] README.rst vs DESCRIPTION.rst

[Marcus Smith]

although I can see the value of distinguishing a description vs readme
file,  I can also see that it's confusing enough to make me want the sample
project to just have a readme for simplicity  (and maybe just mention the
distinction as a possibility)

I opened an issue here  https://github.com/pypa/sampleproject/issues/31

I'd inclined to merge the change if someone posted a PR, or eventually get
to it myself



On Mon, Sep 28, 2015 at 8:18 AM, Chris Barker - NOAA Federal <
chris.bar...@noaa.gov> wrote:

>
>
> Sent from my iPhone
>
> On Sep 27, 2015, at 11:19 AM, Ionel Cristian Mărieș 
> wrote:
>
>
> On Sun, Sep 27, 2015 at 8:05 PM, Thomas Güttler <
> guettliml@thomas-guettler. 
> ​I don't think there can be a "definitive guide line"​. Unlike the core
> language the packaging part of Python is a messy soup of different and
> often competing ideas, styles and tools.
>
>
> Which is EXACTLY why there should be one set of best-practices
> recommendations that are the same in all the "official" docs.
>
> I think one Readme.rst is the way to go.
>
> If you want to provide contribution guidelines, etc. they should be in a
> separate locations, referenced by the Readme.
>
> -Chris
>
>
>
> So you cannot have an definitive or objective guide for something that's
> subjective in nature.
>
> About the README vs DESCRIPTION - ask yourself, what would you use README
> for then? I believe that's absolutely nothing. You only need one. :-)
>
>
>
> Thanks,
> -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] turn down https://bitbucket.org/pypa/pypi-metadata-formats?

[Marcus Smith]

I think we want to turn down
https://bitbucket.org/pypa/pypi-metadata-formats?

Since it's replaced by https://github.com/pypa/interoperability-peps

I'm thinking we should migrate issues (and close the old ones with links to
the new ones), and add a loud notification to the old readme.  People are
still updating and watching issues in the old tracker.

It's confusing.  I'm willing to help here.

Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Marcus Smith]

I'm still unclear on whether you'd want A or B:

A) Different major/minor versions of the spec are different documents
B) Different versions of the spec are tags or branches of the same document

If it's B, then you'd either:
1) only build the latest version, and construct an index of links to the
unrendered old versions in vcs history
2) use a custom build/publishing worflow that pulls versions out of history
so they can be built as peers in the published version





On Sun, Sep 6, 2015 at 9:26 PM, Nick Coghlan <ncogh...@gmail.com> wrote:

> On 7 September 2015 at 14:11, Marcus Smith <qwc...@gmail.com> wrote:
> >
> >
> >> > That way, the URL works as people expect, *and* the resulting
> >> > destination gives a URL that (when inevitably copy-and-pasted) will
> >> > retain its meaning over time.
> >>
> >> Yes, ReadTheDocs does let us do that.
> >
> >
> > well, it lets you do it for a whole project.
>
> RTD also has page redirects now:
>
> https://read-the-docs.readthedocs.org/en/latest/user-defined-redirects.html#page-redirects
> (I thought the same thing you did, but found that when double
> checking)
>
> So we *could* redirect unqualified links to qualified ones if we
> wanted to. I just don't want to :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Marcus Smith]

Wes, this isn't about the versioning scheme for Specs or PEPS.
For *whatever* scheme we have,  my discussion was about how to render all
the "current" versions we support in a Sphinx project.
in short, should the current versions we want to publish be distinct
documents or not.

>  The PEP workflow is probably fine

well, if you look up in the thread, a few of us are saying it's not.  It
doesn't distinguish Current Specs vs Proposals very well.


On Mon, Sep 7, 2015 at 9:40 AM, Wes Turner <wes.tur...@gmail.com> wrote:

> MAJOR.MINOR.PATCH[-rev] would be helpful for these  (and other) PEPs.
>
> On Sep 7, 2015 10:36 AM, "Marcus Smith" <qwc...@gmail.com> wrote:
> >
> > I'm still unclear on whether you'd want A or B:
> >
> > A) Different major/minor versions of the spec are different documents
>
> From http://semver.org Semantic Versioning 2.0 :
>
> ```
> Given a version number MAJOR.MINOR.PATCH, increment the:
>
> - MAJOR version when you make incompatible API changes,
> - MINOR version when you add functionality in a backwards-compatible
> manner, and
> - PATCH version when you make backwards-compatible bug fixes.
>
> Additional labels for pre-release and build metadata are available as
> extensions to the MAJOR.MINOR.PATCH format.
> ```
>
> > B) Different versions of the spec are tags or branches of the same
> document
>
> From http://docs.openstack.org/developer/pbr/semver.html :
>
> ```
> *Linux/Python Compatible Semantic Versioning 3.0.0*
>
> This is a fork of Semantic Versioning 2.0. The specific changes have to do
> with the format of pre-release and build labels, specifically to make them
> not confusing when co-existing with Linux distribution packaging and Python
> packaging. Inspiration for the format of the pre-release and build labels
> came from Python’s PEP440.
>
> *Changes vs **SemVer** 2.0**¶*
> <http://docs.openstack.org/developer/pbr/semver.html#changes-vs-semver-2-0>
>
> dev versions are defined. These are extremely useful when dealing with CI
> and CD systems when ‘every commit is a release’ is not feasible.All
> versions have been made PEP-440 compatible, because of our deep roots in
> Python. Pre-release versions are now separated by . not -, and use a/b/c
> rather than alpha/beta etc.
> ```
>
> Something like v1.0.01-eb4df7f[-linux64] would have greater traceability.
>
> >
> > If it's B, then you'd either:
> > 1) only build the latest version, and construct an index of links to the
> unrendered old versions in vcs history
> > 2) use a custom build/publishing worflow that pulls versions out of
> history so they can be built as peers in the published version
>
> #. TBH I'm more concerned about determining downstream tool support from
> MAJOR.MINOR.PATCH
> (The PEP workflow is probably fine; I think there is need for  better
> versioning under one heading).
>
> >
> >
> >
> >
> >
> > On Sun, Sep 6, 2015 at 9:26 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> >>
> >> On 7 September 2015 at 14:11, Marcus Smith <qwc...@gmail.com> wrote:
> >> >
> >> >
> >> >> > That way, the URL works as people expect, *and* the resulting
> >> >> > destination gives a URL that (when inevitably copy-and-pasted) will
> >> >> > retain its meaning over time.
> >> >>
> >> >> Yes, ReadTheDocs does let us do that.
> >> >
> >> >
> >> > well, it lets you do it for a whole project.
> >>
> >> RTD also has page redirects now:
> >>
> https://read-the-docs.readthedocs.org/en/latest/user-defined-redirects.html#page-redirects
> >> (I thought the same thing you did, but found that when double
> >> checking)
> >>
> >> So we *could* redirect unqualified links to qualified ones if we
> >> wanted to. I just don't want to :)
> >>
> >> Cheers,
> >> Nick.
> >>
> >> --
> >> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> >
> >
> >
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] "Specs" vs "Proposals"

[Marcus Smith]

pulling this idea out of the "Linux wheel support" thread, since it
deserves it's own thread...

the idea being that we should better distinguish:
1) the current packaging "Specs" (for metadata, versions, etc...)
vs
2) Proposals to change them

currently, we just have PEPs that serve both roles.

so the idea would be to:
1) house current specs at packaging.python.org... basically a document tree
that's organized by topic, not numbers and it's free of proposal
rationales, historical discussion, and transition plans etc...
2) keep using the PEP process for adjusting or adding to the specs

and assuming that approach, I raised a few "publishing" questions:
1) do we publish/render all supported versions of a certain spec, or just
the latest
2) if we publish them all, then how?  do we maintain separate documents for
distinct versions? if not, how do we do it?

--Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] "Specs" vs "Proposals"

[Marcus Smith]

> so the idea would be to:
>> 1) house current specs at packaging.python.org... basically a document
>> tree that's organized by topic, not numbers and it's free of proposal
>> rationales, historical discussion, and transition plans etc...
>>
>
> *
> https://packaging.python.org/en/latest/glossary.html#term-version-specifier
> -> pypa.io/en/latest/peps
> * https://www.pypa.io/en/latest/peps
> * https://github.com/pypa/pypa.io/blob/master/source/peps.rst
>
>

I can't make sense of how these links are a response to point #1?
I wrote the PEP summary page you're linking me too, so I'm aware of it  :  )
This summary page is *not* the Specs idea.
Maybe a picture is worth a thousand words here and will require a draft
PR against the PyPUG to make it clear to everyone



> 2) keep using the PEP process for adjusting or adding to the specs
>>
>> and assuming that approach, I raised a few "publishing" questions:
>> 1) do we publish/render all supported versions of a certain spec, or just
>> the latest
>> 2) if we publish them all, then how?  do we maintain separate documents
>> for distinct versions? if not, how do we do it?
>>
>
> Tagged [semver 3.0 (+1)] versions can be managed *individually* with
> readthedocs.
>

sorry, I don't follow your point.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Marcus Smith]

> That way, the URL works as people expect, *and* the resulting
> > destination gives a URL that (when inevitably copy-and-pasted) will
> > retain its meaning over time.
>
> Yes, ReadTheDocs does let us do that.


well, it lets you do it for a whole project.
we'd have to have a project per spec for it to work like that.
we've been talking about all specs being in one project (PyPUG)

I think it we'd either have to:
1) only render the latest version, and construct an index of links to the
unrendered old versions in vcs history
or
2) use a custom tailored tool to publish/render this like we want.
or
3) use distinct documents for distinct versions as peers in the src tree.


-Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] BDFL Delegates for distutils-sig PEPs

[Marcus Smith]

ok, so this is PEP 474
where's the activity for the forge idea happening?   python-dev list?

On Sat, Sep 5, 2015 at 10:47 PM, Nick Coghlan <ncogh...@gmail.com> wrote:

>
> On 6 Sep 2015 10:39, "Marcus Smith" <qwc...@gmail.com> wrote:
> >
> > yea, I like the idea of our own authoritative Pypa project for
> proposals, and maybe have it hold the final "Specs" we're talking about as
> well (and just have PyPUG link to them or whatever).
> >
> > I *think* it would lower the bar for people considering getting involved
> with writing specs and enhancements.
>
> Brett Cannon set an October 31st deadline for the forge.python.org
> proofs-of-concept, so that aspect will be changing regardless (either to a
> PR-capable Kallithea, or GitHub+Phabricator)
>
> Cheers,
> Nick.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] ensurepip in linux distros

[Marcus Smith]

>
>
> then bundles the system
> version back up with rewheel for installation into Python virtual
> environments.


this "bundles the system version back up" step happens when?
which fedora version did this start?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Marcus Smith]

One thought that comes to mind is how to present specs that deal with
formats and artifacts that persist for years.

For example, down the road when there's Wheel 2.0, what's the "Current
Specs" for wheel?

I would describe 2.0 is the "Latest" spec, but "Current Specs" includes all
versions we're attempting to support, so we'd want the "Current Specs" page
to easily show all the versions, and not have to dig them out from version
control or something, right?

--Marcus

On Sat, Sep 5, 2015 at 1:35 AM, Nick Coghlan <ncogh...@gmail.com> wrote:

> On 5 September 2015 at 16:46, Nathaniel Smith <n...@pobox.com> wrote:
> > On Fri, Sep 4, 2015 at 9:24 PM, Marcus Smith <qwc...@gmail.com> wrote:
> >>> I don't have a specific problem with the specs living somewhere else
> >>> as well, I just don't think moving a lengthy document full of edge
> cases
> >>> from one location to another is going to make things better
> >>
> >> If I may, I don't think that really captures Nick's idea.
> >>
> >> I think it's about clearly distinguishing the following:
> >>
> >> 1) Current Specs (for metadata, versioning, pypi etc..)
> >> 2) Proposals to adjust or add to the Current Specs
> >>
> >> We don't have a clear distinction right now.   We just have a series of
> >> PEPs, and it's work to figure out where the actual current spec is at,
> in
> >> the noise of rationales and transition plans etc...
> >
> > Speaking as someone who has been pretty confused in the past trying to
> > look up what the actual current rules are for something like version
> > numbers or metadata (is this the current PEP? oh wait this one's newer
> > -- oh but wait is the newer one still in development? or maybe
> > abandoned?, etc.): +1
>
> We also have specs like Tarek's database of installed distributions
> (https://www.python.org/dev/peps/pep-0376/), where we kept the
> "dist-info" parts, but not any of the API proposals.
>
> *Existing* formats (like sdist) could also be specified there without
> requiring a new PEP (modulo people's time to do the work, but at least
> having a place for such specs to *go* would be a good first step).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] BDFL Delegates for distutils-sig PEPs

[Marcus Smith]

is this a response to other thread about how/where to store specs and PEPs?
If not, what in this email are you responding to?

On Sat, Sep 5, 2015 at 1:32 PM, Donald Stufft  wrote:

> If it’s more useful we could also just use an RFC repository like Rust
> does instead of doing a mishmash between having Python using PEPs and
> packaging using PEPs.
>
> On September 4, 2015 at 11:42:21 PM, Nick Coghlan (ncogh...@gmail.com)
> wrote:
> > We've got to a point where the original standing delegations to myself
> > and Richard Jones to act as BDFL-Delegates for metadata
> > interoperability and pypi.python.org related aren't scaling
> > adequately, so given Paul's recent delegation for PEP 470, and Donald
> > handling PEP 503 directly, it seems like an opportune time to put
> > something in writing about that.
> >
> > For PyPA/distutils-sig specific PEPs, we've effectively adopted the
> > following approach to assigning BDFL-Delegates in resolving PEPs 470
> > and 503:
> >
> > =
> > Whenever a new PEP is put forward on distutils-sig, any PyPA core
> > reviewer that believes they are suitably experienced to make the final
> > decision on that PEP may offer to serve as the BDFL's delegate (or
> > "PEP czar") for that PEP. If their self-nomination is accepted by the
> > other PyPA core reviewer, the lead PyPI maintainer and the lead
> > CPython representative on distutils-sig, then they will have the
> > authority to approve (or reject) that PEP.
> > =
> >
> > And translating the nominated roles to the folks currently filling
> > them: "lead PyPI maintainer" = Donald Stufft; "lead CPython
> > representative on distutils-sig" = me.
> >
> > "PyPA core reviewer" isn't a term we've previously used, but I'm
> > aiming to capture "has approval rights for pull requests to one or
> > more of the PyPA maintained source code or documentation repos".
> >
> > Some further details for the benefit of folks not aware of the relevant
> history:
> >
> > * a couple of years ago, we amended PEP 1 to give the "Discussions-To"
> > header some additional force for PEPs which don't directly affect
> > CPython: """PEP review and resolution may also occur on a list other
> > than python-dev (for example, distutils-sig for packaging related PEPs
> > that don't immediately affect the standard library). In this case, the
> > "Discussions-To" heading in the PEP will identify the appropriate
> > alternative list where discussion, review and pronouncement on the PEP
> > will occur."""
> >
> > * we *didn't* update the section about assignment of BDFL-Delegates.
> > Instead, I received a general delegation for packaging metadata
> > interoperability PEPs, and Richard Jones received one for
> > pypi.python.org related PEPs
> >
> > * Richard subsequently passed the latter delegation on to Donald,
> > since Donald had taken over as the lead maintainer for PyPI
> >
> > The section in PEP 1 for CPython BDFL-Delegates reads as follows:
> > =
> > However, whenever a new PEP is put forward, any core developer that
> > believes they are suitably experienced to make the final decision on
> > that PEP may offer to serve as the BDFL's delegate (or "PEP czar") for
> > that PEP. If their self-nomination is accepted by the other core
> > developers and the BDFL, then they will have the authority to approve
> > (or reject) that PEP.
> > =
> >
> > This process can be appropriately described as "volunteering to be
> > blamed" - if a PEP from a less experienced contributor subsequently
> > proves to be a mistake, that's on the BDFL-Delegate for saying "Yes",
> > not on the PEP author for proposing it. Mostly though, it's so there's
> > someone to have the final say on the fiddly little details that go
> > into getting from a general concept to an actual implementation,
> > without getting mired down in design-by-committee on every incidental
> > detail.
> >
> > As PEP authors, we'll also often ask someone else specifically to
> > volunteer as BDFL-Delegate, because we trust their judgement in
> > relation to the topic at hand (e.g. I asked Martin von Löwis to be
> > BDFL-Delegate for the original ensurepip PEP because I knew he was
> > skeptical of the idea, so a design that passed muster with him was
> > likely to have suitably addressed the ongoing maintainability
> > concerns. Guido did something similar when he asked Mark Shannon to be
> > BDFL-Delegate for PEP 484's gradual typing).
> >
> > Regards,
> > Nick.
> >
> > P.S. It's becoming clear to me that I should probably write a
> > companion PEP to PEP 1 that specifically describes distutils-sig's
> > usage of the PEP process (and how that differs from the normal CPython
> > processes), but hopefully this post provides sufficient clarification
> > for now.
> >
> > --
> > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
> > 

Re: [Distutils] BDFL Delegates for distutils-sig PEPs

[Marcus Smith]

yea, I like the idea of our own authoritative Pypa project for proposals,
and maybe have it hold the final "Specs" we're talking about as well (and
just have PyPUG link to them or whatever).

I *think* it would lower the bar for people considering getting involved
with writing specs and enhancements.

"Oh, it's just a github project, and you do PRs... I can do that... let me
start writing now..."

for ensurepip,  yea, do PEPs, since it's in Python.



On Sat, Sep 5, 2015 at 4:49 PM, Donald Stufft <don...@stufft.io> wrote:

> On September 5, 2015 at 7:13:18 PM, Nick Coghlan (ncogh...@gmail.com)
> wrote:
> > On 6 Sep 2015 08:31, "Marcus Smith" wrote:
> > >
> > > is this a response to other thread about how/where to store specs and
> > PEPs?
> > > If not, what in this email are you responding to?
> >
> > I believe Donald was suggesting we could just have a PyPA specific change
> > proposal process hosted on packaging.python.org, rather than using a
> > variant of the PEP process.
> >
> > I don't want to do that though - PyPA/distutils-sig's authority *is*
> > delegated from python-dev through the BDFL-Delegate and Discussions-To
> > headers in the regular PEP process, and there are going to be some
> > proposals affecting ensurepip and distutils that still fall under
> > python-dev rather than distutils-sig.
> >
> > Dealing with the PEP repo is currently more painful than it needs to be,
> > but that's what the forge.python.org proposals aim to address.
> >
>
>
> I was yes, though I don't feel extremely strongly about it, but I do
> wonder if
> it wouldn't fit us better. I'll make my case here real quick, but unless
> others
> are interested in it I don't really feel strong enough to push for it.
>
> We're currently using the PEP process, but we've had to bend the PEP rules
> several times in order to fit us. It's obvious the PEP process is designed
> primarily to handle changes to Python itself, even the BDFL-Delegate rules
> currently state you have to be a Python core developer and that you need
> permission from Guido for it. On top of that, almost all of the things we
> touch
> don't really fall under python-dev's authority. PyPI, pip, setuptools,
> bandersnatch, devpi, etc are external projects and only really distutils
> and
> ensurepip need python-dev's stamp of approval. In a way, we're already
> part way
> to the RFC process that rust uses through the interoperability-peps repo on
> Github. The primary differences would be that we manually copy things over
> to
> the Python PEPs repository and we don't really follow the rules for
> BDFL-Delegates, we just invent our own rules and it's sort of OK because
> nobody
> really cares.
>
> It is kind of awkward to essentially have the "real" copy of the PEP live
> in
> Github and that's where all the work on it happens but then needing to
> manually
> copy things over. It makes it kind of annoying to work on things,
> especially
> since any typo change or the such requires additional work to keep the two
> copies in sync.
>
> On the other hand, if we focused on a process that worked for us, instead
> of
> trying to shoe horn things into the PEP process we could optimize it for
> how
> we function. This could also include direct integration with
> packaging.python.org or something similar if we wanted to go down that
> road.
>
> I don't know, I think we could have a better process if we did our own
> thing,
> so it's something to think about at least.
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Marcus Smith]

> I don't have a specific problem with the specs living somewhere else
> as well, I just don't think moving a lengthy document full of edge cases
> from one location to another is going to make things better

If I may, I don't think that really captures Nick's idea.

I think it's about clearly distinguishing the following:

1) Current Specs (for metadata, versioning, pypi etc..)
2) Proposals to adjust or add to the Current Specs

We don't have a clear distinction right now.   We just have a series of
PEPs, and it's work to figure out where the actual current spec is at, in
the noise of rationales and transition plans etc...

- So, for #1, maintain documents in PyPUG
- For #2, keep using PEPs
- As PEPs are accepted, update the Spec docs (the version history can
mention what PEP drove the change)

And separate from all of this I think is your idea that regular Usage docs
should be modified as well, as PEPs are accepted, which I think is great.

Marcus




On Fri, Sep 4, 2015 at 8:06 PM, Donald Stufft  wrote:

> On September 4, 2015 at 10:56:38 PM, Nick Coghlan (ncogh...@gmail.com)
> wrote:
> > On 5 September 2015 at 12:14, Donald Stufft wrote:
> > > On September 4, 2015 at 10:12:08 PM, Nick Coghlan (ncogh...@gmail.com)
> wrote:
> > >> It's only the interoperability specs where we currently follow the RFC
> > >> model of having the same document describe both the end result *and*
> > >> the rationale for changes from the previous version, and I mostly find
> > >> it to be a pain.
> > >>
> > >
> > > I'm not sure that I follow what you’re saying here, can you describe
> what your
> > > ideal situation would look like?
> >
> > 1. We add a new section to packaging.python.org for "Specifications".
> > The specification sections of approved PEPs (compatibility tags, wheel
> > format, version specifiers, dist-info directories) get added there.
> > API specifications for index servers may also be added there.
> >
> > 2. Interoperability PEPs become proposals for new packaging.python.org
> > specifications or changes to existing specifications, rather than
> > specifications in their own right.
> >
> > 3. Each specification has a "version history" section at the bottom,
> > which links to the PEPs that drove each update.
> >
> > This way, the PEPs can focus on transition plans, backwards
> > compatibility constraints, and the rationale for particular changes,
> > etc, but folks wanting "just the current spec, thanks" can look at the
> > latest version on packaging.python.org without worrying about the
> > history.
> >
> > It also means that the specs themselves (whether additions or updates)
> > can be prepared as packaging.python.org PRs.
> >
>
> Personally I don't have much of a problem with the specs living as PEPs, I
> think a bigger problem is that we're producing specs that have end user
> impact
> without anything designed for end users to go along with them. PEP 440 is a
> wonderful example of this, the spec of PEP 440 goes into lots of edge
> cases and
> describes the reasons why particular decisions were made and all of that
> jazz.
> I think all of that data is useful when you're implementing PEP 440
> because it
> helps inform how someone interprets the spec in situations where it may be
> ambiguous.
>
> What I don't think is useful is having no answer to someone who asks
> "What's
> a valid version for a Python package" except "here go read this massive
> document which covers tons of edge cases which you don't really need to
> care
> about unless you're pip/PyPI/setuptools etc".
>
> I guess for me then, the ideal situation would be to keep using PEPs for
> the
> actual specification/RFC like documentation, but when that has end user
> impact
> then a requirement is that it comes with a PR to packaging.python.org that
> gives us end user documentation for the spec, before the spec can be
> accepted
> (or finalized or whatever the right terminology is). I mean, I don't have a
> specific problem with the specs living somewhere else as well, I just don't
> think moving a lengthy document full of edge cases from one location to
> another
> is going to make things better unless we start producing end user focused
> documentation, and in many cases it may make it worse since understanding a
> spec fully often requires understanding why certain decisions were made.
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] ensurepip in linux distros

[Marcus Smith]

Can anyone summarize the state of ensurepip for the major linux distros.

Do any currently include a version that leaves ensurepip intact?
If not, will any? Moreover, would any ever also bootstrap pip for you?

I'm not asking out of interest in wanting it, more to understand for the
sake of editing pip and PyPUG docs.

thanks,
Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] mention Copr/EPEL/IUS in pip install instructions?

[Marcus Smith]

Hello:

I'm looking for opinions on mentioning Copr and/or EPEL and/or IUS in the
pip install instructions.

Here's the PR with the actual docs changes:
https://github.com/pypa/pip/pull/3067

The goal is to give people a linux distro-friendly way (at least for
fedora/centos/rhel) to upgrade pip (in distro-managed pythons) to a newer
version,  i.e. something more friendly than "get-pip.py"

I started the thread in the pypa-dev list.   thought I might get broader
feedback over here.

feedback in the PR is appreciated.

thanks
Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Marcus Smith]

I think Linux wheel support is almost useless unless the pypa stack
 provides _something_ to handle non-python dependencies.


I wouldn't say useless, but I tend to agree with this sentiment.

I'm thinking the only way to really compete with the ease of Conda (for
non-python dependencies) is to shift away from wheels, and instead focus on
making it easier to create native distro packages (i.e. rpm, deb etc...that
can easily depend on non-python dependencies) for python applications, and
moreover that these packages should be parallel installable with the
system packages, i.e. they should depend on virtual environments, not the
system python.

I've been working on this some personally, admittedly pretty slowly, since
it's a pretty tall order to put all the pieces together

Marcus



 1) Pure Python packages work fine as source.

 2) Python packages with C extensions build really easily out of the box
 -- so source distribution is fine (OK, I suppose some folks want to
 run a system without a compiler -- is this the intended use-case?)

 So what are the hard cases? the one we really want binary wheels for?

  - Windows, where a system compiler is a rarity: Done

  - OS-X, where a system compiler is a semi-rarity, and way too many
 standard system libs aren't there (or are old crappy versions...) -
 Almost Done.

  - Packages with semi-standard dependencies: can we expect ANY Linux
 distro to have libfreetype, libpng, libz, libjpeg, etc? probably, but maybe
 not installed (would a headless server have libfreetype?). And would those
 version be all compatible (probably if you specified a distro version)

  - Packages with non-standard non-python dependencies: libhdf5, lapack,
 BLAS, fortran(!) -- this is where the nightmare really is. I suspect most
 folks on this list will say that this is Scipy Problem, and indeed,
 that's where the biggest issues are, and where systems like conda have
 grown up to address this.

 But at this point, I think it's really sad that the community has become
 fractured -- if folks start out with I want to do scientific computing,
 then they get pointed to Enthought Canopy or Anaconda, and all is well
 (until they look for standard web development packages -- though that's
 getting better). But if someone starts out as a web developer, and is all
 happy with the PyPA stack (virtualenv, pip, etc...), then someone suggests
 they put some Bokeh plotting in their web site, or need to do
 some analytics on HDF5 files, or any number of things well supported by
 Python, but NOT by pip/wheel -- they are kind of stuck.

 My point is that it may actually be a bad thing to solve the easy problem
 while keeping out fingers in our ears about the hard ones

 (la la la la, I don't need to use those packages. la  la la la)

 My thought: what pip+wheel needs to support much of this is the ability to
 specify a wheel dependency, rather than a package dependency -- i.e. this
 particular wheel requires a libfreetype wheel. Then we could have binary
 wheels for non-python dependencies like libs (which would install the lib
 into pre-defined locations that could be relative path linked to)

 Sorry for the rant

 -Chris

 PS: Personally, after banging my head against this for years,
 I've committed to conda for the moment -- working to get conda to better
 support the wide range of python packages. I haven't tried it on Linux, but
 it does exist and works well for some folks.



 On Fri, Jul 17, 2015 at 1:22 AM, Nick Coghlan ncogh...@gmail.com wrote:

 On 17 July 2015 at 03:41, Nate Coraor n...@bx.psu.edu wrote:
  Hi all,
 
  I've recently been working on adding SOABI support for Python 2.x and
 other
  pieces needed to get wheels w/ C extensions for Linux working. Here's
 the
  work for wheels:
 
  https://bitbucket.org/pypa/wheel/pull-request/54/
 
  Based on that, I've added support for those wheels to pip here:
 
  https://github.com/natefoo/pip/tree/linux-wheels

 
  As mentioned in the wheels PR, there are some questions and decisions
 made
  that I need guidance on:
 
  - On Linux, the distro name/version (as determined by
  platform.linux_distribution()) will be appended to the platform string,
 e.g.
  linux_x86_64_ubuntu_14_04. This is going to be necessary to make a
  reasonable attempt at wheel compatibility in PyPI. But this may violate
 PEP
  425.

 I think it's going beyond it in a useful way, though. At the moment,
 the linux_x86_64 platform tag *under*specifies the platform - a
 binary extension built on Ubuntu 14.04 with default settings may not
 work on CentOS 7, for example.

 Adding in the precise distro name and version number changes that to
 *over*specification, but I now think we can address that through
 configuration settings on the installer side that allow the
 specification of compatible platforms. That way a derived
 distribution could add the corresponding upstream distribution's
 platform tag and their users would be able to install the relevant
 wheel 

Re: [Distutils] PyPI is a sick sick hoarder

[Marcus Smith]

Why not start with pip at least being a simple fail-on-conflict resolver
(vs the 1st found wins resolver it is now)...

You'd backtrack for the sake of re-walking when new constraints are
found, but not for the purpose of solving conflicts.

I know you're motivated to solve Openstack build issues, but many of the
issues I've seen in the pip tracker, I think would be solved without the
backtracking resolver you're trying to build.

On Fri, May 15, 2015 at 11:57 AM, Robert Collins robe...@robertcollins.net
wrote:

 So, I am working on pip issue 988: pip doesn't resolve packages at all.

 This is O(packages^alternatives_per_package): if you are resolving 10
 packages with 10 versions each, there are approximately 10^10 or 10G
 combinations. 10 packages with 100 versions each - 10^100.

 So - its going to depend pretty heavily on some good heuristics in
 whatever final algorithm makes its way in, but the problem is
 exacerbated by PyPI's nature.

 Most Linux (all that i'm aware of) distributions have at most 5
 versions of a package to consider at any time - installed(might be
 None), current release, current release security updates, new release
 being upgraded to, new release being upgraded to's security updates.
 And their common worst case is actually 2 versions: installed==current
 release and one new release present. They map alternatives out into
 separate packages (e.g. when an older soname is deliberately kept
 across an ABI incompatibility, you end up with 2 packages, not 2
 versions of one package). To when comparing pip's challenge to apt's:
 apt has ~20-30K packages, with altnernatives ~= 2, or
 pip has ~60K packages, with alternatives ~= 5.7 (I asked dstufft)

 Scaling the number of packages is relatively easy; scaling the number
 of alternatives is harder. Even 300 packages (the dependency tree for
 openstack) is ~2.4T combinations to probe.

 I wonder if it makes sense to give some back-pressure to people, or at
 the very least encourage them to remove distributions that:
  - they don't support anymore
  - have security holes

 If folk consider PyPI a sort of historical archive then perhaps we
 could have a feature to select 'supported' versions by the author, and
 allow a query parameter to ask for all the versions.

 -Rob

 --
 Robert Collins rbtcoll...@hp.com
 Distinguished Technologist
 HP Converged Cloud
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Idea: move accepted interoperability specifications to packaging.python.org

[Marcus Smith]

the PyPA site has a PEP reference that includes details on implementation:

https://www.pypa.io/en/latest/peps

I don't think we need another reference in the Packaging User Guide
(PUG).   We could mention that the PyPA one exists in the PUG.

As for user-facing PEP docs, I think the docs for PUG/pip/setuptools/wheel
etc.. should handle that inline, and act as the layer that references (if
need be) the correct PEP that applies to the feature being described.

For example, the PUG references PEP440 in a few places.

Ideally, it should be a requirement to update the docs for the major
projects (and the PUG) *before* releasing a PEP-implementations into those
projects.

For example, a round of docs PRs prior to the release of PEP440 into pug 
pip  could have likely prevented the confusion over the meaning  of 
that unfortunately had to be sorted out after the release.


On Sun, Apr 19, 2015 at 1:21 AM, holger krekel hol...@merlinux.eu wrote:


 I'd appreciate a current packaging specs site which ideally also states
 how pypa tools support it, since which version.

 holger

 On Fri, Apr 17, 2015 at 16:18 -0400, Nick Coghlan wrote:
  Daniel's started work on a new revision of the wheel specification,
  and it's crystallised a concern for me that's been building for a
  while: the Python Enhancement Proposal process is fundamentally a
  *change management* process and fundamentally ill-suited to acting as
  a *hosting service* for the resulting reference documentation.
 
  This is why we're seeing awkward splits like the one I have in PEP
  440, where the specification itself is up top, with the rationale for
  changes below, and the large amounts of supporting material in PEP
  426, where the specification is mixed in with a lot of background and
  rationale that isn't relevant if you just want the technical details
  of the latest version of the format.
 
  It also creates a problem where links to PEP based reference documents
  are fundamentally unstable - when we define a new version of the wheel
  format in a new PEP then folks are going to have to follow the daisy
  chain from PEP 427 through to the new PEP, rather than having a stable
  link that automatically presents the latest version of the format,
  perhaps with archived copies of the old version readily available.
 
  I think I see a way to resolve this, and I believe it should be fairly
  straightforward: we could create a specifications section on
  packaging.python.org, and as we next revise them, we start migrating
  the specs themselves out of the PEP system and into
  packaging.python.org. This would be akin to the change in the Python
  3.3, where the specification of the way the import system worked
  finally moved from PEP 302 into the language reference.
 
  Under that model, the wheel 2.0 would be specifically focused on
  describing and justifying the *changes* between 1.0 and 2.0, but the
  final spec itself would be a standalone document living on
  packaging.python.org, and prominently linked to from both PEP 427
  (which it would Supersede) and from the new PEP.
 
  This approach also gives a much nicer model for fixing typos in the
  specifications - those would just be ordinary GitHub PR's on the
  packaging.python.org repo, rather than needing to update the PEPs
  repo.
 
  Thoughts?
 
  Regards,
  Nick.
 
  --
  Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
  ___
  Distutils-SIG maillist  -  Distutils-SIG@python.org
  https://mail.python.org/mailman/listinfo/distutils-sig
 

 --
 about me:http://holgerkrekel.net/about-me/
 contracting: http://merlinux.eu
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] name of the dependency problem

[Marcus Smith]

 level 0 A
   A level 1  1.4= C


 level 0 B
   B level 1  1.6= C 1.7

 pip manages to download version 1.8 of C(Django) using A's requirement,
 but never even warns us that the B requirement of C was violated. Surely
 even in the absence of a resolution pip could raise a warning at the end.



agreed on the warning, but there is a documented workaround for this, that
is to put the desired constraint for C at level 0 (i.e. in the install args
or in your requirements file)

see case #2 here
https://pip.pypa.io/en/latest/user_guide.html#requirements-files
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] name of the dependency problem

[Marcus Smith]

On Wed, Apr 15, 2015 at 8:56 AM, Robin Becker ro...@reportlab.com wrote:

 On 15/04/2015 16:49, Marcus Smith wrote:
 ..


 agreed on the warning, but there is a documented workaround for this, that
 is to put the desired constraint for C at level 0 (i.e. in the install
 args
 or in your requirements file)

 see case #2 here
 https://pip.pypa.io/en/latest/user_guide.html#requirements-files

  indeed pushing all the requirements to level 0 is a way to solve the
 dependency problem myself.


well, not all, just the cases where an override to the default first
found, wins routine, is needed
again, I agree a warning would be a great,  just noting that people can
work around this (once they realize there's a problem), without having to
manually order things (as you mentioned in your last post)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires for dev environments

[Marcus Smith]

 For instance, if the problem is when setuptools does the install, then
 things
 get installed differently, with different options, SSL certs, proxies, etc
 then I think a better solution is that pip does terrible hacks in order to
 forcibly take control of setup_requires from setuptools and installs them
 into
 a temporary directory (or something like that). That is something that
 would
 require no changes on the part of authors or people installing software,
 and
 is backwards compatible with everything that's already been published using
 setup_requires.


Donald, could you add a pip issue for the forcibly take control idea (if
we don't have one already?)
this comes up a fair amount, and it would be nice to be able to link to
this.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires for dev environments

[Marcus Smith]

 So you *can* import things inside of a setup.py today, you just have
to

I think it's time for the Packaging User Guide to try to cover
setup_requires...
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting more momentum for pip

[Marcus Smith]

So I guess my suggestions boil down to:


 - Add more humans
 - Add more money to make humans more efficient
 - Add more computer automation



maybe agree to always maintain  X open issues and  Y open PRs, before
adding features.
where x can vary as needed, but for starters, x=250, and y=25 sounds
reasonable.
this would:
  - force more work on issue and PR backlog
  - force making the tough decisions on whether something is realistically
going to be worked on and closing it
  - force closing issues that are not getting the responses needed to
actually work the problem.
  - would more likely cycle in new folks to become committers.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting more momentum for pip

[Marcus Smith]

 Currently there are no labels at all for any issue or PR.



there are labels  https://github.com/pypa/pip/labels
I put most of these last year.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting more momentum for pip

[Marcus Smith]

On Thu, Mar 5, 2015 at 10:21 AM, Randy Syring ra...@thesyrings.us wrote:



  On 03/05/2015 12:07 PM, Paul Moore wrote:

 It seems to me that there is another point that delays progress on a
 certain proportion of PRs, specifically feature requests, namely that
 no-one really has that strong an opinion on whether they are a good
 idea or not. I know I've noticed a few requests recently where my
 reaction was essentially I don't have a problem with this, but I
 don't care enough to actually add it


 An issue before feature request PR policy might help here.  Before
 someone submits the feature request PR, have them open an issue for
 discussion of the feature.  Recommend/Require that at least one core dev be
 strongly in favor before giving approval to open the PR.  Automatically
 close these issues without the approval after some amount of inactivity.
 Automatically lose any feature request PRs that don't have an approved
 issue associated with them.

 Put up an explanation of this policy, sympathy that it may cause
 frustration, but that with limited time, devs must focus their efforts, etc.


+1.  good idea.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting more momentum for pip

[Marcus Smith]

 That implies closing 183 issues and 65 PRs from where we are now. And
 when you say adding features presumably that means somehow
 forbidding people (core devs? we can't forbid anyone else...) from
 creating new PRs until we're below the limit.

 In general, I don't think it's practical in a volunteer-based project,
 to force people to hit specific targets.



you certainly can't be hard line about it, but I think it's a practical
rule and achievable.
you just don't work on new non-critical things until you get the issue and
PR counts down.
As much as I would like to see pypa have a system of issue and PR
review,  I think a simple rule like this is more achievable right now.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parsing requirements, pip has no API ...

[Marcus Smith]

 In general, requirements.txt seems to be an
 anti-pattern. You either have to use likely to break tooling or you'll
 have to reinvent that from scratch. You're better off putting it
 directly in setup.py and using setup.py to install dependencies in a
 virtualenv instead of requirements.txt


I don't know your context for calling it an anti-pattern, but there are
valid use cases for requirements.txt vs install_requires.
here's what the Python Packaging User Guide has on the distinction

https://packaging.python.org/en/latest/requirements.html

skipping to the distinctions, it lists four:

Whereas install_requires defines the dependencies for a single
project, *Requirements
Files* https://pip.pypa.io/en/latest/user_guide.html#requirements-files
are often used to define the requirements for a complete python environment.
Whereas install_requires requirements are minimal, requirements files often
contain an exhaustive listing of pinned versions for the purpose of
achieving *repeatable installations*
https://pip.pypa.io/en/latest/user_guide.html#repeatability of a complete
environment.

Whereas install_requires requirements are “Abstract”, requirements files
often contain pip options like --index-url or --find-links to make
requirements “Concrete”. [1]
https://packaging.python.org/en/latest/requirements.html#id4

Whereas install_requires metadata is automatically analyzed by pip during
an install, requirements files are not, and only are used when a user
specifically installs them using pip install -r.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parsing requirements, pip has no API ...

[Marcus Smith]

In 90% of the cases I see, requirements.txt are used to define the

 requirements for the project to function which typically are the exact
 same requirements necessary when installing the project. People also
 will then write a test-requirements.txt (or dev-requirements.txt) file
 to have a complete development environment. In this case, Thomas seems
 to be using requirements.txt to define the requirements necessary when
 installing the software they're developing.

 If requirements.txt were used solely for a development environment,
 they would look more like

 | .
 | dev-requirement-1=0.1
 | # etc.

 Instead they seem to be used more to define the same requirements
 someone would define in install_requires. This is the anti-pattern I'm
 talking about.


understood, I hear you, but for a breakdown of some valid use cases, the
pip docs covers 4 good use cases

https://pip.pypa.io/en/latest/user_guide.html#requirements-files
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Idea: allow PyPI projects to link to DockerHub container images

[Marcus Smith]

 As a possible interim approach to improving the situation, what do you
 think of my writing up a Binary distribution for Linux advanced
 topic? That could cover not only containers, but also the technique of
 bundle a /opt virtualenv in a platform binary package as well as
 actually creating native system packages (with varying degrees of
 distro policy compliance).


sounds great to me
fwiw, I have a project *idea* going to address this problem as well,
although it's still vaporware right now.

http://pyospkg.readthedocs.org/en/latest/overview.html
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Idea: allow PyPI projects to link to DockerHub container images

[Marcus Smith]

 I agree that from an implementation perspective, this could just be a
 new recommended URL in the project URLs metadata (e.g. Reference
 Container Images). If folks don't think the idea sounds horrible,
 I'll make that update to the PEP 459 draft.


wouldn't this just be a use case for a custom Metadata Extension, and not
something new to put into PEP459?
I'm just imagining you won't cover everything that might be needed for
automation down the road, and it will end up being an unused field in a few
years.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPA Announcements Twitter

[Marcus Smith]

+1 to a pypa-announce list.
I personally care more about the list than twitter.
at this point, probably need to post the idea to pypa-dev, and get a few
+1's there, and get someone to agree to execute on the idea.
there's a few people there that don't monitor distutils-sig
Marcus


On Sun, Jan 25, 2015 at 12:33 PM, Marc Abramowitz msabr...@gmail.com
wrote:


  On Jan 25, 2015, at 11:45 AM, Randy Syring ra...@thesyrings.us wrote:
 
  Therefore, I'd like to propose something like a packaging-announce
 mailing list.  This could be linked to the PyPA Twitter account.  Then,
 announcements that pertain to the broader Python community would be cross
 posted to the distutils-sig and packaging-announce (and therefore twitter).

 +1 to this.

 I think the key observation here is that there are two audiences when it
 comes to packaging:

 1. A large audience of folks who are package authors and maintainers, who
 want to know about the latest developments and best practices.

 2. The smaller audience of folks who know packaging well and are working
 hard to improve it and having discussions among themselves on how to best
 move things forward.

 I don't know of an information channel that serves #1 well. The ones I
 know of try to handle both. I think having different channels for each
 group will improve the experience for both groups.

 Randy, I'm willing to help if you want.

 Marc
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] problem viewing pep 440 docs on mobile

[Marcus Smith]

yes, I see a similar bug on android.
If I try to scroll up a *little*, it goes all the back to the top.

On Fri, Jan 2, 2015 at 7:39 AM, Chris Jerdonek chris.jerdo...@gmail.com
wrote:

 Sorry if this isn't the best list on which to bring this up, but it came
 up for me during the recent PEP 440 discussions.

 For a while I've noticed a serious problem when viewing PEP doc pages like
 the following on my iPhone:

 https://www.python.org/dev/peps/pep-0440/

 It's a bit maddening, so I wanted to see if anyone else on this list has
 experienced it or can reproduce it.

 It happens for me with both Chrome and Safari on the latest iOS 8.1.2,
 though I believe it happened for me on older iOS versions, too.

 After the page is fully loaded, if I scroll down, say, halfway through the
 page, the browser will jump back to the very top.  This effectively makes
 it impossible to read the page.  I haven't noticed this issue with any page
 not hosted on python.org.

 Can anyone else reproduce?

 Thanks,
 --Chris


 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Marcus Smith]

  but look at this (using setuptools 8)
 
  '1.7.dev1' in pkg_resources.Requirement.parse('foo=1.7')
  False
  '1.7.dev1' in pkg_resources.Requirement.parse('foo=1.7')
  True

 I believe the first example is a consequence of the following two
 excerpts from the PEP [...] In both cases (by (2) above), you

are supposed to compare the
 candidate version 1.7.dev1 with the specified version 1.7.


yes, I agree with you.  :  )
my 2 examples weren't meant to exemplify the inconsistency I was referring
to.
they were intended to be response to you quoting the bit about implicitly
excluded
Because the 2nd example was True, they're not implicitly excluded.

For fear that my current point is lost now (since it's changed from the
original title), I'm going to start a new thread that sums it up real
brief...

Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP440: 1.7 vs =1.7

[Marcus Smith]

I'm starting a new thread to state cleanly what my current question/concern
is...

per PEP440 as I understand it:
- for 1.7, 1.7 means roughly the 1.7 series or 1.7*
- for =1.7, 1.7 means literally 1.7 (with zero-padding as needed)

While I understand the motivation for the series conception of 1.7 to
deal with  prereleases, the resulting inconsistency in meaning for 1.7 is
what concerns me.  It's odd .  Can someone make it not seem odd for 1.7 to
change meanings just because you switch between = and ?

And for anyone who wants to say that 1.7 also means the 1.7 series for
=, note that 1.7 prereleases do not satisfy =1.7

Would a solution be to make = also use the series concept?, i.e. make
pre-releases satisfy =1.7

Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: 1.7 vs =1.7

[Marcus Smith]

 In gives me a minor bit of pause. However any alternative that I can come
 up
 with bothers me more, especially since I don't believe many people actually
 even *use* a bare  and any alternative I can come up with has worse
 behavior
 for operators which get much more use.


what about making =,= also use the series concept? where does that go
wrong?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Marcus Smith]

Thinking of 1.7 as greater than the 1.7 series sort of helps me as
 well...



the problem with thinking of it this way is that you naturally want to
extend the concept to =, but it doesn't work.
If the concept were consistent,  1.7.dev1 would satisfy  =1.7, but it
doesn't.
for =, the concept seems to be zero-padding.
for , the concept is the series wildcard idea.

this inconsistency concerns me, and makes it hard to edit the PEP to be
more understandable like I would have wanted.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Marcus Smith]

  the problem with thinking of it this way is that you naturally want to
  extend the concept to =, but it doesn't work.
  If the concept were consistent,  1.7.dev1 would satisfy  =1.7, but it
  doesn't.

 I'm pretty sure it's consistent.  For example, 1.7.2 doesn't satisfy
 1.7, but it satisfies =1.7 (e.g. because it's part of the
 series).

 I believe the inconsistency you're mentioning doesn't have anything to
 do with the comparison operator.  It's that pre-releases like dev1
 are special cases and governed by different rules. [...] are
 implicitly excluded


but look at this (using setuptools 8)

 '1.7.dev1' in pkg_resources.Requirement.parse('foo=1.7')
False
 '1.7.dev1' in pkg_resources.Requirement.parse('foo=1.7')
True

Only the exclusive ordering section speaks of the != V.* bit.
The inclusive section just talks of zero padding and relative ordering.

The packaging implementation that underlies all this seems to bear this out
as well:

inclusive:
https://github.com/pypa/packaging/blob/master/packaging/specifiers.py#L453
exclusive:
https://github.com/pypa/packaging/blob/master/packaging/specifiers.py#L461
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Marcus Smith]

 It *should* be being kept in sync with the published versions, but Donald
 and I have just been working directly in the main PEP repo recently.

 Hence my suggestion of moving it to GitHub - it's more likely to be kept
 up to date there, and, unlike the master PEP repo, I'm happy to host the
 working drafts and issue tracker for the metadata PEPs wherever folks find
 them most convenient to work with.

I'd suggest pypa/peps on github (i.e. a more general name than
pypi-metadata-formats)
and ideally change the name of the pep src file after it has a pep number.
(i.e. from versioning.rst to pep0440.rst)
also, maybe isolate the named drafts (with no number) in a drafts folder.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Role of setuptools and eggs in modern distributing...

[Marcus Smith]

 A note about terminology here (both in this email and The Packaging User
 Guide) -- it seems to me that install_requires is about requirements for a
 package not a project,


well, read through the PyPUG glossary:
https://packaging.python.org/en/latest/glossary.html

a project is anything with an associated setup.py (which will often have
install_requires metadata)

a package (listed as a distribution package in the glossary) is the
distribution of a certain release of a project



 I guess the distinction may be that a package has a setup.py, whereas a
 project is somethign you are building that requires perhaps a stack of
 unrelated packages.



Above, I used the word environment, which was just short hand for the
whole set of installed packages on the Python path for the interpreter used
by your application.   This is often literally a virtual environment
created by virtualenv.

To me, the distinction is over which project *owns* the whole environment,
i.e  what is the top-level project that the environment exists for.

Requirements files are typically associated with the project that owns the
environment.


Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Role of setuptools and eggs in modern distributing...

[Marcus Smith]

 git+https://url_to_the_repo.git#egg=name_of_package

 why isn't that wheel=name_of_package


the egg part here has nothing to do with eggs.  just a vestige of another
time.
see  https://github.com/pypa/pip/issues/1265

and will it work if setuptools was not used in the packages setup.py???


yes, it would work.

which I can now get with pip --editable or  does that give me setuptools
 develop mode anyway


-e uses setuptools develop mode.


So question one: do I need to use setuptools.setup rather than plain old
 distutils.setup?


the main reason for setuptools is for install_requires, which is
fundamental to pip dependency resolution.
but in general, it offers more features and it's more maintained than pure
distutils.
The standard advice is to use setuptools over distutils.

I generally like the pip requirements.txt approach. It's up to the
 installation tool, not the packaging tool to mange requirements. But then
 again, it does make sense to declare the requirements in setup.py. But the
 issue at hand is that install_requires is doing some odd things with conda:


The Packaging User Guide has a breakdown of install_requires vs
requirements files.
https://packaging.python.org/en/latest/technical.html#install-requires-vs-requirements-files

In brief, requirements files are usually for a whole environment, whereas
install_requires is for a project.

install_requires is critical when publishing projects to PyPI.

Even if you're not publishing, install_requires is helpful to safely
attempt an upgrade of your dependencies.
Instead of installing your app from a frozen requirements file, you would
install normally in a clean environment (based on install_requires) and let
any upgrades occur, and then you can re-freeze your requirements.

But if you are doing mostly vcs installs in your requirements file, then
it's true that maintaining install_requires can be somewhat pointless,
although I would still say it's helpful to keep track of each project's
first level abstract dependencies are, even if the install_requires
declarations aren't being used.

Marcus
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Marcus Smith]

just post edit suggestions here?

https://bitbucket.org/pypa/pypi-metadata-formats  is not up to date anymore

some other location?

On Tue, Dec 23, 2014 at 1:01 AM, Donald Stufft don...@stufft.io wrote:


 On Dec 23, 2014, at 1:23 AM, Marcus Smith qwc...@gmail.com wrote:


 In particular, , , ~=, and, when using a .*, the != and == use the
 number of dots in the given specifier to indicate the precision of the
 specifier.


 the PEP text is pretty clear on the precision concept for ~= and when
 using .*, but not so much for  and .
 how about an example right there in that section that mentions the !=
 V.*


 We can certainly clarify the PEP test to make it easier to understand.

 ---
 Donald Stufft
 PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440 examples missing operator?

[Marcus Smith]

 Heh, that’s actually a hold over from before we made specifiers mandatory.
 That needs updated.


you mean made operators mandatory?
so the bit about version identifier without any comparison operator needs
to be removed?



 ---
 Donald Stufft
 PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Marcus Smith]

it would fail.  you'd need  1.7.0

On Mon, Dec 22, 2014 at 12:36 PM, James Bennett ubernost...@gmail.com
wrote:

 So, if PyPI has foo-1.7 and foo-1.7.1, does 1.7 just fail to find
 anything installable?

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Marcus Smith]

This makes sense if you view 1.7 as encompassing all of the releases

 1.7.0, 1.7.1, 1.7.2, etc. (and their variants).  It's a bit like the
 significant digits of a number. [1]  You want the release to be
 bigger than all releases of that type.


but is this really the underlying concept?
Does the PEP explain this as a justification? Where?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Marcus Smith]

 * The new behavior maintains consistency between  and , so that
 specifiers that “look” the same act the same, maintaining consistency
 between them.
 * I think that having the  and  behavior vary is a *worse* confusion,
 and I believe that the behavior of  is far better than previous.


I'm not following the old inconsistencies you're referring to. Maybe you
can explain those, or reference this in the PEP.
But in any case, the argument isn't over whether PEP440 is better than the
old implementation, in total.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Marcus Smith]

 By the way, note that 1.7 doesn't *always* mean 1.7.*


right, got it
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP440 examples missing operator?

[Marcus Smith]

the first 3 examples here are confusing.
https://www.python.org/dev/peps/pep-0440/#examples

there's no comparison operator?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440 examples missing operator?

[Marcus Smith]

oh, A compatible release clause consists of either a version identifier
without any comparison operator or else the compatible release operator ~=

On Mon, Dec 22, 2014 at 9:57 PM, Marcus Smith qwc...@gmail.com wrote:

 the first 3 examples here are confusing.
 https://www.python.org/dev/peps/pep-0440/#examples

 there's no comparison operator?

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440: foo-X.Y.Z does not satisfy fooX.Y?

[Marcus Smith]

 In particular, , , ~=, and, when using a .*, the != and == use the
 number of dots in the given specifier to indicate the precision of the
 specifier.


the PEP text is pretty clear on the precision concept for ~= and when using
.*, but not so much for  and .
how about an example right there in that section that mentions the != V.*
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposal: using /etc/os-release in the platform tag definition for wheel files

[Marcus Smith]

+1 to the idea in general.
Would this be an *edit* to PEP425/427/Wheel-1.0  OR new peps, and a new
wheel version?
As someone using cent6 daily (with no os-release file), I'm greedy for
another fallback technique, but the simplicity of just using os-release
makes sense.
Could a published linux_x86_64_fedora_20 wheel ever become broken just
due to normal yum update activity on fedora_20?  When? Why?

On Thu, Nov 27, 2014 at 10:03 PM, Nick Coghlan ncogh...@gmail.com wrote:

 We've discussed the idea of changing the wheel file naming scheme to
 deal with Linux previously, but never put together a concrete
 proposal.

 The closest we've got is the idea of allowing the platform tag to be
 customised in pip and perhaps bdist_wheel, and while that's good from
 an enabling experimentation perspective, it may be overkill if the
 primary goal is just to better support handling of Linux distros.

 For starters, here's the current definition of the platform tag in PEP 425:

 =
 The platform tag is simply distutils.util.get_platform() with all
 hyphens - and periods . replaced with underscore _ .

 * win32
 * linux_i386
 * linux_x86_64
 =

 Here's my proposed change:

 =
 The default platform tag is distutils.util.get_platform() with all
 hyphens - and periods . replaced with underscore _ . If
 /etc/os-release [N] exists on the system, then the values in the 'ID'
 and 'VERSION_ID' fields are read, all hyphens - and periods . replaced
 with underscore _ , and the results appended to the default tag after
 a separating underscore.

 Examples:

 * win32
 * macosx_10_6_intel
 * linux_x86_64_fedora_20
 * linux_x86_64_rhel_7_0
 * linux_x86_64_debian_7_0
 * linux_x86_64_ubuntu_14_04
 =

 The [N] reference would then be a reference to
 http://www.freedesktop.org/software/systemd/man/os-release.html for
 the definition of the format of os-release. (Note that while the
 format originated with systemd, plenty of distros have also started
 providing it regardless of which init system they use)

 Now, this slightly overspecifies on the *consumer* side. A binary
 wheel that works on rhel_7_0 for example, should almost certainly
 work on rhel_7_1. However, that can be addressed on the tooling side
 (e.g. permitting the specification of additional compatible
 platforms when invoking pip), rather than needing to be in the
 specification.

 This also won't help with older Linux distros that don't offer
 /etc/os-release, but I'm OK with that - those can just continue to
 show up as linux_x86_64, and PyPI can continue to disallow those
 uploads.

 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP425 - Wheel Binary Package Compatibility

[Marcus Smith]

 This is the kind of direction we're also exploring for Fedora  EPEL:
 setting up a distro specific devpi instance so we can automatically publish
 distro compatible wheel files, as well as separating the distro level
 licencing and preliminary security review step from the step of repackaging
 in a language independent format. (I need to set up a curated PyPI mirror
 for work anyway, and since I work at Red Hat, and the Fedora community are
 open to the idea, we're working on it upstream rather than inside the
 firewall)


Btw, I've been doing this at my company, that is maintaining separate wheel
indexes per platform we need (in our case, cent5 and cent6).
One problem here is that it's possible for pip download caches to get
corrupted with the wrong distributions (i.e. the incompatible ones),
since the distributions are not distinguishable by name
Would Fedora really consider releasing a public service where there fedora
release is not represented in the distribution name? or would you be
wanting to get more tagging spec'd out in Wheel 2.0 first?

P.S.  In looking at the PEP425/427 specs again, I'm slightly confused about
the build tag.  It's barely covered, but I can imagine using that
possibly to distinguish dists, assuming pip support was added, alhough it's
supposed to start with a number, which doesn't seem to fit the use case
we'd want.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP425 - Wheel Binary Package Compatibility

[Marcus Smith]

yes, I'm partial to a solution like this prior to wheel 2.0 (that I imagine
would support additional/custom tags)

On Wed, Oct 29, 2014 at 1:46 PM, Daniel Holth dho...@gmail.com wrote:

 I've always thought people might like to have a custom platform tag or
 otherwise customize the supported list in a configuration file loaded
 by pep425tags. Then you could create wheels called
 somewheel-4.0-py33-none-30caa8a209d6.whl (choose your own random
 string). Only a pip with a matching config would pay any attention to
 those wheels. That way we could help people manage their internal
 deployments without falsely advertising that we've solved the binary
 incompatibility feature baked into every Linux distribution.

 On Wed, Oct 29, 2014 at 3:52 PM, Donald Stufft don...@stufft.io wrote:
 
  On Oct 29, 2014, at 2:36 PM, Marcus Smith qwc...@gmail.com wrote:
 
 
  This is the kind of direction we're also exploring for Fedora  EPEL:
  setting up a distro specific devpi instance so we can automatically
 publish
  distro compatible wheel files, as well as separating the distro level
  licencing and preliminary security review step from the step of
 repackaging
  in a language independent format. (I need to set up a curated PyPI
 mirror
  for work anyway, and since I work at Red Hat, and the Fedora community
 are
  open to the idea, we're working on it upstream rather than inside the
  firewall)
 
 
  Btw, I've been doing this at my company, that is maintaining separate
 wheel
  indexes per platform we need (in our case, cent5 and cent6).
  One problem here is that it's possible for pip download caches to get
  corrupted with the wrong distributions (i.e. the incompatible ones),
 since
  the distributions are not distinguishable by name
  Would Fedora really consider releasing a public service where there
 fedora
  release is not represented in the distribution name? or would you be
 wanting
  to get more tagging spec'd out in Wheel 2.0 first?
 
  P.S.  In looking at the PEP425/427 specs again, I'm slightly confused
 about
  the build tag.  It's barely covered, but I can imagine using that
 possibly
  to distinguish dists, assuming pip support was added, alhough it's
 supposed
  to start with a number, which doesn't seem to fit the use case we'd want.
 
 
  ___
  Distutils-SIG maillist  -  Distutils-SIG@python.org
  https://mail.python.org/mailman/listinfo/distutils-sig
 
 
  It’s a build number, it’s used incase you need to rebuild a Wheel using
 the
  same source files.
 
  ---
  Donald Stufft
  PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
 
 
  ___
  Distutils-SIG maillist  -  Distutils-SIG@python.org
  https://mail.python.org/mailman/listinfo/distutils-sig
 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig