Re: [Distutils] What's special about numpy, scipy, ...was: Remove distutils, was: red, green, refactor ...

[Thomas Güttler]

Am 26.10.2015 um 20:07 schrieb Nathaniel Smith:
> On Oct 26, 2015 6:59 AM, "Wayne Werner"  > wrote:
>  >
>  > On Thu, Oct 22, 2015 at 11:27 AM, Chris Barker  > wrote:
>  >>
>  >> On Thu, Oct 22, 2015 at 8:47 AM, Thomas Güttler 
> > wrote:
>  >>
>  >>>
>  >>> > I have a dream: For packaging and building package provides only 
> **data**. Data is condition-less: No single "if", "def" or method call. Just 
> data: json or yaml ...
>  >>> >
>  >>> > Even for complex packages.
>  >>
>  >>
>  >> It's a nice dream -- but I think there is more or less a consensus on 
> this list that you really can't do everything completely declaratively. I 
> know I would find it very frustrating to have to hard-code everything up 
> front.
>  >
>  >
>  > I've seen this sentiment mentioned several times... Having only had 
> experience with sdist-style packages which are dead simple to define, do you 
> have any examples of some specific thing that's a pain to do declaratively?
> 
> Test the system's implementation of the math.h 'tanh' function to decide 
> whether it correctly implements C99 annex F style handling of edge cases and 
> thus can be used safely, or whether we need to fall back to our internal 
> version of tanh instead, and define some appropriate C preprocessor macros 
> depending on the result.


With a plugin-system it would be easy to handle this declaratively.

We need a way to extend the supported key-value pairs. This would be done in 
check_tanh plugin package.

Then you need to build-depend on the new check_tanh package.

I guess that most developers feel that this is very complicated. I feel the 
same, but I **think** it is not.
The structure is straight forward. It just feels complicated. The check_tanh 
package
can be create in just a few minutes.


Regards,
  Thomas Güttler


-- 
http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wacky idea about reifying extras

[David Cournapeau]

On Mon, Oct 26, 2015 at 11:41 PM, Nathaniel Smith  wrote:

> On Mon, Oct 26, 2015 at 4:41 AM, Donald Stufft  wrote:
> > On October 26, 2015 at 3:36:47 AM, Nathaniel Smith (n...@pobox.com)
> wrote:
> >> > TL;DR
> >> -
> >>
> >> If we:
> >>
> >> - implement a real resolver, and
> >> - add a notion of a per-project namespace of distribution names,
> >> that
> >> are collected under the same PyPI registration and come from
> >> the same
> >> sdist, and
> >> - add Conflicts:, and Provides:,
> >>
> >> then we can elegantly solve a collection of important and difficult
> >> problems, and we can retroactively pun the old extras system
> >> onto the
> >> new system in a way that preserves 100% compatibility with all
> >> existing packages.
> >>
> >> I think?
> >>
> >> What do you think?
> >
> > My initial reaction when I started reading your idea was that I didn't
> see a
> > point in having something like foo[bar] be a "real" package when you
> could just
> > as easily have foo-bar. However, as I continued to read through the idea
> it
> > started to grow on me. I think I need to let it percolate in my brain a
> little
> > bit, but there may be a non-crazy (or at least, crazy in a good way)
> idea here
> > that could push things forward in a nice way.
>
> Oh good, at least I'm not the only one :-).
>
> I'd particularly like to hear Robert's thoughts when he has time,
> since the details depend strongly on some assumptions about how a real
> resolver would work.
>
> > Some random thoughts:
> >
> > * Reusing the extra syntax is nice because it doesn't require end users
> to
> >   learn any new concepts, however we shouldn't take a new syntax off the
> table
> >   either if it makes the feature easier to implement with regards to
> backwards
> >   compatability. Something like numpy{mkl,some-other-thing} could work
> just as
> >   well too. We'll need to make sure that whatever symbols we choose can
> be
> >   represented on all the major FS we care about and that they are
> ideally non
> >   ugly in an URL too. Of course, the filename and user interface symbols
> don't
> >   *need* to match. It could just as easily example numpy[mkl] out to
> numpy#mkl
> >   or whatever which should make it easier to come up with a nice scheme.
>
> Right -- obviously it would be *nice* to keep the number of concepts
> down, and to avoid skew between filenames and user interface (because
> users do see filenames), but if these turn out to be impossible then
> there are still options that would let us save the
> per-project-package-namespace idea.
>
> > * Provides is a bit of an odd duck, I think in my head I've mostly come
> to
> >   terms with allowing unrestricted Provides when you've already
> installed the
> >   package doing the Providing but completely ignoring the field when
> pulling
> >   data from a repository. Our threat model assumes that once you've
> selected to
> >   install something then it's generally safe to trust (though we still
> do try
> >   to limit that). The problem with Provides mostly comes into play when
> you
> >   will respect the Provides: field for any random package on PyPI (or
> any other
> >   repo).
>
> Yeah, I'm actually not too worried about malicious use either in
> practice, for the reason you say. But even so I can think of two good
> reasons we might want to be careful about stating exactly when
> "Provides:" can be trusted:
>
> 1) if you have neither scipy nor numpy installed, and you do 'pip
> install scipy', and scipy depends on the pure virtual package
> 'numpy[abi-2]' which is only available as a Provides: on the concrete
> package 'numpy', then in this case the resolver has to take Provides:
> into account when pulling data from the repo -- if it doesn't, then
> it'll ignore the Provides: on 'numpy' and say that scipy's
> dependencies can't be satisfied. So for this use case to work, we
> actually do need to be able to sometimes trust Provides: fields.
>
> 2) the basic idea of a resolver is that it considers a whole bunch of
> possible configurations for your environment, and picks the
> configuration that seems best. But if we pay attention to different
> metadata when installing as compared to after installation, then this
> skew makes it possible for the algorithm to pick a configuration that
> looks good a priori but is broken after installation. E.g. for a
> simple case:
>
>   Name: a
>   Conflicts: some-virtual-package
>
>   Name: b
>   Provides: some-virtual-package
>
> 'pip install a b' will work, because the resolver ignores the
> Provides: and treats the packages as non-conflicting -- but then once
> installed we have a broken system. This is obviously an artificial
> example, but creating the possibility of such messes just seems like
> the kind of headache we don't need. So I think whatever we do with
> Provides:, we should do the same thing both before and after
> installation.
>

Another simple solution for this particular case is to 

Re: [Distutils] Second draft of a plan for a new source tree / sdist format

[Paul Moore]

On 26 October 2015 at 06:04, Nathaniel Smith  wrote:
> Here's a second round of text towards making a build-system
> independent interface between pip and source trees/sdists. My idea
> this time is to take a divide-and-conquer approach: this text tries to
> summarize all the stuff that it seemed like we had mostly reached
> consensus on in the previous thread + call, with blank chunks marked
> "TBD" where there are specific points that still need To Be
> Determined. So my hope is that everyone will read what's here and
> agree that it's great as far as it goes, and then we can go through
> and fill in each missing piece one at a time.

I'll comment on what's here, but ignore the TBD items - I'd rather (as
you suggest) leave discussion of those details till the basic idea is
agreed.

> Abstract
> 
>
> Distutils delenda est.

While this makes a nice tagline, I'd rather something less negative.
Distutils does not "need" to be destroyed. It's perfectly adequate
(although hardly user friendly) for a lot of cases - I'd be willing to
suggest *most* users can work just fine with distutils.

I'm not a fan of distutils, but I'd prefer it if we kept the rhetoric
limited - as Nick pointed out this whole area is as much a political
issue as a technical one.

> Extended abstract
> =
>
> While ``distutils`` / ``setuptools`` have taken us a long way, they
> suffer from three serious problems: (a) they're missing important
> features like autoconfiguration and usable build-time dependency
> declaration, (b) extending them is quirky, complicated, and fragile,
> (c) it's very difficult to use anything else, because they provide the
> standard interface for installing python packages expected by both
> users and installation tools like ``pip``.

Again, this is overstated. You very nearly lost me right here - people
won't read the details of the proposal if they disagree with the
abstract(s). Specifically:

* The features in (a) are only important to *some* parts of the
community. The scientific community is the major one, and is a huge
influence over the direction we want to go in, but again, not crucial
to many people. And even where they might be useful (e.g., Windows
users building pyyaml, lxml, pillow, ...) the description implies
"working out what's there" rather than "allowing users to easily
manage non-Python dependencies", which gives the wrong impression.

* The features in (b) are highly specialised. Very few people extend
setuptools/distutils. And those who do, have often invested a lot of
effort in doing so. Sure, they'd rather not have needed to, but now
that they have, a replacement system simply means that work is lost.
Arguably, fixing (b) is only useful for people (like the scientific
community) who have needed to extend setuptools and have been unable
to achieve their goals that way. That's an even smaller part of the
community.

> Previous efforts (e.g. distutils2 or setuptools itself) have attempted
> to solve problems (a) and/or (b). We propose to solve (c).

Agreed - this is a good approach. But it's at odds with your abstract,
which says distutils must die. Here you're saying you want to allow
people to keep using distutils but allow people with specialised needs
to choose an alternative. Or are you offering an alternative to people
who use distutils?

The whole of the above is confusing on the face of it. The details
below clarify a lot, as does knowing how the previous discussions have
gone. But it would help a lot if the introduction to this PEP were
clearer.

> The goal of this PEP is get distutils-sig out of the business of being
> a gatekeeper for Python build systems. If you want to use distutils,
> great; if you want to use something else, then that should be easy to
> do using standardized methods. The difficulty of interfacing with
> distutils means that there aren't many such systems right now, but to
> give a sense of what we're thinking about see `flit
> `_ or `bento
> `_. Fortunately, wheels have now
> solved many of the hard problems here -- e.g. it's no longer necessary
> that a build system also know about every possible installation
> configuration -- so pretty much all we really need from a build system
> is that it have some way to spit out standard-compliant wheels.

OK. Although I see a risk here that if I want to build package FOO, I
now have to worry whether FOO's build system supports Windows, as well
as worrying whether FOO itself supports Windows.

There's still a role for some "gatekeeper" (not a good word IMO, maybe
"coordinator") to provide a certain level of support or review of
build systems, and a point of contact for users with build issues (the
point of this proposal is to some extent that people don't need to
*know* what build system a project uses, so suggesting everyone has to
direct issues to the correct build system support forum isn't
necessarily 

Re: [Distutils] wacky idea about reifying extras

[Robert Collins]

On 27 October 2015 at 21:47, David Cournapeau  wrote:

> Another simple solution for this particular case is to add conflict rules
> between packages that provide the same requirement (that's what php's
> composer do IIRC).
>
> The case of safety against malicious forks is handled quite explicitly in
> composer, we may want to look at how they do it when considering solutions
> (e.g. https://github.com/composer/composer/issues/2690, though it has
> changed a bit since then)
>
> Adding the provides/conflict concepts to pip resolver will complexify it
> quite significantly, both in terms of running time complexity (since at that
> point you are solving a NP-complete problem) and in terms of implementation.
> But we also know for real cases this is doable, even in pure python
> (composer handles all the cases you are mentioning, and is in pure php).

We already require a full NP-complete solver because of <, <= and ~
version operators.

I haven't adsorbed this proposal enough to comment on the reification
aspect yet.

I'm worried about provides and conflicts in general, but not from a
resolver code perspective - thats a one-ish-time-cost, but instead
from a user experience perspective.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Second draft of a plan for a new source tree / sdist format

[Daniel Holth]

The drawback of .zip is file size since it compresses each file
individually rather than giving the compression algorithm a larger input,
it's a great format otherwise. Ubiquitous including Apple iOS packages,
Java, word processor file formats. And most Python packages are small.

We must do the hard work to support Unicode file names, and spaces and
accent marks in home directory names (historically a problem on Windows),
in our packaging system. It is the right thing to do. It is not the
publisher's fault that your system has broken Unicode.

On Tue, Oct 27, 2015 at 6:43 AM Paul Moore  wrote:

> On 26 October 2015 at 06:04, Nathaniel Smith  wrote:
> > Here's a second round of text towards making a build-system
> > independent interface between pip and source trees/sdists. My idea
> > this time is to take a divide-and-conquer approach: this text tries to
> > summarize all the stuff that it seemed like we had mostly reached
> > consensus on in the previous thread + call, with blank chunks marked
> > "TBD" where there are specific points that still need To Be
> > Determined. So my hope is that everyone will read what's here and
> > agree that it's great as far as it goes, and then we can go through
> > and fill in each missing piece one at a time.
>
> I'll comment on what's here, but ignore the TBD items - I'd rather (as
> you suggest) leave discussion of those details till the basic idea is
> agreed.
>
> > Abstract
> > 
> >
> > Distutils delenda est.
>
> While this makes a nice tagline, I'd rather something less negative.
> Distutils does not "need" to be destroyed. It's perfectly adequate
> (although hardly user friendly) for a lot of cases - I'd be willing to
> suggest *most* users can work just fine with distutils.
>
> I'm not a fan of distutils, but I'd prefer it if we kept the rhetoric
> limited - as Nick pointed out this whole area is as much a political
> issue as a technical one.
>
> > Extended abstract
> > =
> >
> > While ``distutils`` / ``setuptools`` have taken us a long way, they
> > suffer from three serious problems: (a) they're missing important
> > features like autoconfiguration and usable build-time dependency
> > declaration, (b) extending them is quirky, complicated, and fragile,
> > (c) it's very difficult to use anything else, because they provide the
> > standard interface for installing python packages expected by both
> > users and installation tools like ``pip``.
>
> Again, this is overstated. You very nearly lost me right here - people
> won't read the details of the proposal if they disagree with the
> abstract(s). Specifically:
>
> * The features in (a) are only important to *some* parts of the
> community. The scientific community is the major one, and is a huge
> influence over the direction we want to go in, but again, not crucial
> to many people. And even where they might be useful (e.g., Windows
> users building pyyaml, lxml, pillow, ...) the description implies
> "working out what's there" rather than "allowing users to easily
> manage non-Python dependencies", which gives the wrong impression.
>
> * The features in (b) are highly specialised. Very few people extend
> setuptools/distutils. And those who do, have often invested a lot of
> effort in doing so. Sure, they'd rather not have needed to, but now
> that they have, a replacement system simply means that work is lost.
> Arguably, fixing (b) is only useful for people (like the scientific
> community) who have needed to extend setuptools and have been unable
> to achieve their goals that way. That's an even smaller part of the
> community.
>
> > Previous efforts (e.g. distutils2 or setuptools itself) have attempted
> > to solve problems (a) and/or (b). We propose to solve (c).
>
> Agreed - this is a good approach. But it's at odds with your abstract,
> which says distutils must die. Here you're saying you want to allow
> people to keep using distutils but allow people with specialised needs
> to choose an alternative. Or are you offering an alternative to people
> who use distutils?
>
> The whole of the above is confusing on the face of it. The details
> below clarify a lot, as does knowing how the previous discussions have
> gone. But it would help a lot if the introduction to this PEP were
> clearer.
>
> > The goal of this PEP is get distutils-sig out of the business of being
> > a gatekeeper for Python build systems. If you want to use distutils,
> > great; if you want to use something else, then that should be easy to
> > do using standardized methods. The difficulty of interfacing with
> > distutils means that there aren't many such systems right now, but to
> > give a sense of what we're thinking about see `flit
> > `_ or `bento
> > `_. Fortunately, wheels have now
> > solved many of the hard problems here -- e.g. it's no longer necessary
> > that a build system also know about 

[Distutils] (no subject)

[Maciej Szulik]

Thanks to Nick Coghlan and Barry Warsaw we've setup a new SIG dedicated
to discussing python features from different distributions point of view.

Here is Nick's reasoning:
> With the Python 3 migration, and the growth in interest in user level
> package management for development purposes, what do you think of the idea
> of setting up a new Linux SIG to have those discussions? I know it's a
case
> of "yet another mailing list", but I think it will be worthwhile to have a
> clear point of collaboration within the Python ecosystem, rather than
> expecting Pythonistas to know how to reach out to (other) distros
directly.

The list is available @ https://mail.python.org/mailman/listinfo/linux-sig


Maciej
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP

[Daniel Holth]

It looks like you would normally be able to say just 'flit' in the
bootstrap-requires or provided-by, unless the requirements were for build
code bundled inside the sdist.

On Tue, Oct 27, 2015 at 1:07 AM Robert Collins 
wrote:

> On 27 October 2015 at 14:06, Robert Collins 
> wrote:
>
> > We can rev this in schema version 2. There's no prose in the PEP about
> > how that should work, so I'll add that now.
>
> Done: https://gist.github.com/rbtcollins/666c12aec869237f7cf7#upgrades
>
> -Rob
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Time for a setuptools_lite??

[Daniel Holth]

Mr. grumpy pants,

Now you're trolling us by describing flit.

Older versions of pip could be able to install it, just add a setup.py
shim. Nearly trivial.

A new simple build system could come with the agile build system support.
It's just a matter of timing.

On Sun, Oct 25, 2015 at 9:49 PM Donald Stufft  wrote:

> On October 25, 2015 at 7:17:12 PM, Nick Coghlan (ncogh...@gmail.com)
> wrote:
> >
> > And now that I've got my "Mr grumpy pants" reaction to the earlier
> > parts of the thread out of my system (sorry about that), I'd like to
> > revisit Chris's idea of having a *simple* replacement for setuptools
> > as a build system.
> >
>
> I’m not *against* it, but it has some problems:
>
> * Older versions of pip and setuptools will not be able to install it at
> all. This includes anyplace setuptools is used implicitly such as
> ``setup.py install`` from another package that depends on it or
> setup_requires or test_requires etc.
>
> * A lot of downstream folks will probably be mad at us for introducing a
> second way of doing things (particularly since it won’t come with the agile
> build system support that we have on the road map that will make this a lot
> better for non setuptools build systems).
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Second draft of a plan for a new source tree / sdist format

[Paul Moore]

On 27 October 2015 at 13:12, Daniel Holth  wrote:
> We must do the hard work to support Unicode file names, and spaces and
> accent marks in home directory names (historically a problem on Windows), in
> our packaging system. It is the right thing to do. It is not the publisher's
> fault that your system has broken Unicode.

In the examples I'm thinking of, the publisher used a format (.tar.gz)
that didn't properly support Unicode, in the sense that it didn't
include an encoding for the bytes it used to represent filenames. IMO,
that is something we shouldn't allow, by rejecting file formats that
don't support Unicode properly.

Whose fault it is, is not important - it's just as easy to say that
it's not the end user's fault that the publisher made an unwarranted
assumption about encodings. What's important is that things work for
everyone, and the interoperability standards don't leave room for
people to make such assumptions.

Paul

PS Consider this a retraction of my suggestion that filenames in
sdists should be pure ASCII. But still, sdists shouldn't contain files
that can't be used on target systems - e.g., 2 files whose names
differ only in case, files containing characters like :, ? or * that
are invalid on Windows... Whether this needs to be noted in the
standard, or whether it's just a case of directing users' bug reports
back to the publisher, is an open question, though.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wacky idea about reifying extras

[Brett Cannon]

On Tue, 27 Oct 2015 at 02:17 Robert Collins 
wrote:

> On 27 October 2015 at 21:47, David Cournapeau  wrote:
>
> > Another simple solution for this particular case is to add conflict rules
> > between packages that provide the same requirement (that's what php's
> > composer do IIRC).
> >
> > The case of safety against malicious forks is handled quite explicitly in
> > composer, we may want to look at how they do it when considering
> solutions
> > (e.g. https://github.com/composer/composer/issues/2690, though it has
> > changed a bit since then)
> >
> > Adding the provides/conflict concepts to pip resolver will complexify it
> > quite significantly, both in terms of running time complexity (since at
> that
> > point you are solving a NP-complete problem) and in terms of
> implementation.
> > But we also know for real cases this is doable, even in pure python
> > (composer handles all the cases you are mentioning, and is in pure php).
>
> We already require a full NP-complete solver because of <, <= and ~
> version operators.
>
> I haven't adsorbed this proposal enough to comment on the reification
> aspect yet.
>
> I'm worried about provides and conflicts in general, but not from a
> resolver code perspective - thats a one-ish-time-cost, but instead
> from a user experience perspective.
>

So from my perspective as someone who (I think) grasps what the problems
that everyone is trying to solve is but not knowing enough to know how
stuff is done now (all my projects on PyPI are pure Python), Nathaniel's
proposal makes total sense to me.

I would think it would be easy to explain to a scientist that "to get
scipy, run `python3.5 -m pip install numpy`, but if you want fast over open
source and use Intel's MKL library, do `python3.5 -m pip install
numpy[mkl]`. I think the syntax clearly shows it's a
modification/tweak/special version of numpy and it makes sense that I want
to install something that provides numpy while relying on MKL.

Nathaniel's comment about how this might actually give pip a leg up on
conda also sounds nice to me as I have enough worry about having a fissure
in 1D along the Python 2/3 line, and I'm constantly worried that the
scientific community is going to riot and make it a 2D fissure along Python
2/3, pip/conda axes and split effort, documentation, etc.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP

[Nathaniel Smith]

On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins
 wrote:
> Since Nathaniel seems busy, I've taken the liberty of drafting a
> narrow PEP based on the conversations that arose from the prior
> discussion.
>
> It (naturally) has my unique flavor, but builds on the work Nathaniel
> had put together, so I've put his name as a co-author even though he
> hasn't seen a word of it until now :) - all errors and mistakes are
> therefore mine...
>
> Current draft text in rendered form at:
> https://gist.github.com/rbtcollins/666c12aec869237f7cf7
>
> I've run it past Donald and he has a number of concerns - I think
> we'll need to discuss them here, and possibly in another hangout, to
> get a path forward.

Now that I've had a chance to read it properly...

First impression: there's surprisingly little overlap between this and
my simultaneously-posted draft [1] -- my draft focuses on trying to
only document the stuff that everyone seemed to agree on, includes a
proposal for static metadata in sdists (since Donald seemed to be
saying that he considered this a mandatory component of any proposal
to update how sdists work), and tries to set out a blueprint for how
to organize the remaining issues, whereas yours spends most of its
time on the controversial details that I decided to skip over for this
draft.

Being an engineer, there are a number of these details that I'm
tempted to quibble over :-), but I will restrain myself for now and
wait first to see how the other thread develops and whether we do have
general agreement on the points I wrote, since this seems like a
complex enough topic that it could easily spiral become a
conversational morass if we aren't careful to keep the different
issues organized?

-n

[1] https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP

[Marcus Smith]

>
>
> Current draft text in rendered form at:
> https://gist.github.com/rbtcollins/666c12aec869237f7cf7
>
>
Thanks for working on this.
Overall I like the idea, but have some comments/questions


1)  *Please*, *please*, *please* let's start doing PEP conversations as PRs
to pypa/interoperability-peps :  )  There's a place in there for unnumbered
PEPs.  Review will be better and faster IMO as PRs.  If the PR gets too
cluttered with old conversations, just start clean with a new PR.   We can
announce the PRs here, but don't discuss them on the mailing list.   The
same goes for trimming down PEP426 as you mention.   Let's do that as a PR
to pypa/interoperability-peps to the existing document.

2) Ditto on Ralf's concerns about readability.  I only really understood it
after seeing the examples you gave to Ralf (they need to be in the PEP, not
just in response to Ralf).   There's a few places I'd want to alter the
wording, but I likely won't bother, unless it's done as a PR.

3) You refer to "source distribution".  What does that mean exactly in this
PEP?  just the current setuptools notion?

4) Although using a process interface is not necessarily a problem, I don't
agree with your point on why a python interface would be unworkable.
You're assuming that pip would try to import all the build tools (for every
dependency it's processing) in the master process.  An alternative could be
that pip would have it's own tool (that runs as a subprocess in an isolated
env) that knows how to load and work with python build interfaces.   You
could argue that a python api is  an advantage, because build tools aren't
forced to grow a certain cli for pip, they just have to add a shim module
that conforms to the interface.   But my point is not to argue for the
Python API, but for you to remove an argument that from what I can tell,
isn't really an argument for one or the other.

5) at one pt you said "--dump-build-requires would just output a constant
string: {"schema_version": "2.0", "build_requires": []}"   you meant
"metadata_version", right?

6)  Can you explain better why we need to manage a pypa.yaml schema version
*and* a build description schema version.   Why not just a version for
pypa.yaml, and if anything changes (in the yaml schema or the build
"description" api),  then just bump the version.

7) it's unclear when pip get's to run "dist-info" and when the result might
be different.   For example, we've discussed that run time dependencies may
get clarifed *after* the build process so this command might produce
different results at different times?

--Marcus



> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Please don't impose additional barriers to participation (was: build system abstraction PEP)

[Ben Finney]

Marcus Smith  writes:

> 1) *Please*, *please*, *please* let's start doing PEP conversations as
> PRs to pypa/interoperability-peps : )

Please keep the conversation on a mailing list where one can participate
without needing to sign up with any particular service provider.

Your proposal would have the effect of excluding people from the
conversation if they don't agree to have a GitHub account. I think it's
valuable to avoid that barrier to entry, needing only an email account.

-- 
 \   “'Tis strange, — but true; for truth is always strange; / |
  `\Stranger than fiction.” —“Lord” George Gordon Noel Byron, _Don |
_o__)Juan_ |
Ben Finney

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP

[Ralf Gommers]

On Wed, Oct 28, 2015 at 6:03 AM, Nathaniel Smith  wrote:

> On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins
>  wrote:
> > Since Nathaniel seems busy, I've taken the liberty of drafting a
> > narrow PEP based on the conversations that arose from the prior
> > discussion.
> >
> > It (naturally) has my unique flavor, but builds on the work Nathaniel
> > had put together, so I've put his name as a co-author even though he
> > hasn't seen a word of it until now :) - all errors and mistakes are
> > therefore mine...
> >
> > Current draft text in rendered form at:
> > https://gist.github.com/rbtcollins/666c12aec869237f7cf7
> >
> > I've run it past Donald and he has a number of concerns - I think
> > we'll need to discuss them here, and possibly in another hangout, to
> > get a path forward.
>
> Now that I've had a chance to read it properly...
>
> First impression: there's surprisingly little overlap between this and
> my simultaneously-posted draft [1] --


Which is good, double work has been kept to a minimum - it's like you two
actually coordinated this:)


> my draft focuses on trying to
> only document the stuff that everyone seemed to agree on, includes a
> proposal for static metadata in sdists (since Donald seemed to be
> saying that he considered this a mandatory component of any proposal
> to update how sdists work), and tries to set out a blueprint for how
> to organize the remaining issues, whereas yours spends most of its
> time on the controversial details that I decided to skip over for this
> draft.
>

Imho they're not details. The controversial parts of your draft are still
mostly in the metadata part. If you'd split your draft in two, then you'd
see that the first one is pretty short and the second half of it is only
TBDs. And those TBDs are exactly what Robert's draft fills in.

@Robert: thanks for the example, very helpful. I'll look at it in more
detail later.

Ralf
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Second draft of a plan for a new source tree / sdist format

[David Cournapeau]

On Tue, Oct 27, 2015 at 1:12 PM, Daniel Holth  wrote:

> The drawback of .zip is file size since it compresses each file
> individually rather than giving the compression algorithm a larger input,
> it's a great format otherwise. Ubiquitous including Apple iOS packages,
> Java, word processor file formats. And most Python packages are small.
>

I don't really buy the indexing advantages, especially w/ the current
implementation of zipfile in python (e.g. loading the whole set of archives
at creation time)

A common way to solve the fast metadata access from archive is to archive
separately the metadata data and data (e.g. a zipfile containing 2
zipfiles, one being the original sdist, the other one containing the
metadata).

David


> We must do the hard work to support Unicode file names, and spaces and
> accent marks in home directory names (historically a problem on Windows),
> in our packaging system. It is the right thing to do. It is not the
> publisher's fault that your system has broken Unicode.
>
> On Tue, Oct 27, 2015 at 6:43 AM Paul Moore  wrote:
>
>> On 26 October 2015 at 06:04, Nathaniel Smith  wrote:
>> > Here's a second round of text towards making a build-system
>> > independent interface between pip and source trees/sdists. My idea
>> > this time is to take a divide-and-conquer approach: this text tries to
>> > summarize all the stuff that it seemed like we had mostly reached
>> > consensus on in the previous thread + call, with blank chunks marked
>> > "TBD" where there are specific points that still need To Be
>> > Determined. So my hope is that everyone will read what's here and
>> > agree that it's great as far as it goes, and then we can go through
>> > and fill in each missing piece one at a time.
>>
>> I'll comment on what's here, but ignore the TBD items - I'd rather (as
>> you suggest) leave discussion of those details till the basic idea is
>> agreed.
>>
>> > Abstract
>> > 
>> >
>> > Distutils delenda est.
>>
>> While this makes a nice tagline, I'd rather something less negative.
>> Distutils does not "need" to be destroyed. It's perfectly adequate
>> (although hardly user friendly) for a lot of cases - I'd be willing to
>> suggest *most* users can work just fine with distutils.
>>
>> I'm not a fan of distutils, but I'd prefer it if we kept the rhetoric
>> limited - as Nick pointed out this whole area is as much a political
>> issue as a technical one.
>>
>> > Extended abstract
>> > =
>> >
>> > While ``distutils`` / ``setuptools`` have taken us a long way, they
>> > suffer from three serious problems: (a) they're missing important
>> > features like autoconfiguration and usable build-time dependency
>> > declaration, (b) extending them is quirky, complicated, and fragile,
>> > (c) it's very difficult to use anything else, because they provide the
>> > standard interface for installing python packages expected by both
>> > users and installation tools like ``pip``.
>>
>> Again, this is overstated. You very nearly lost me right here - people
>> won't read the details of the proposal if they disagree with the
>> abstract(s). Specifically:
>>
>> * The features in (a) are only important to *some* parts of the
>> community. The scientific community is the major one, and is a huge
>> influence over the direction we want to go in, but again, not crucial
>> to many people. And even where they might be useful (e.g., Windows
>> users building pyyaml, lxml, pillow, ...) the description implies
>> "working out what's there" rather than "allowing users to easily
>> manage non-Python dependencies", which gives the wrong impression.
>>
>> * The features in (b) are highly specialised. Very few people extend
>> setuptools/distutils. And those who do, have often invested a lot of
>> effort in doing so. Sure, they'd rather not have needed to, but now
>> that they have, a replacement system simply means that work is lost.
>> Arguably, fixing (b) is only useful for people (like the scientific
>> community) who have needed to extend setuptools and have been unable
>> to achieve their goals that way. That's an even smaller part of the
>> community.
>>
>> > Previous efforts (e.g. distutils2 or setuptools itself) have attempted
>> > to solve problems (a) and/or (b). We propose to solve (c).
>>
>> Agreed - this is a good approach. But it's at odds with your abstract,
>> which says distutils must die. Here you're saying you want to allow
>> people to keep using distutils but allow people with specialised needs
>> to choose an alternative. Or are you offering an alternative to people
>> who use distutils?
>>
>> The whole of the above is confusing on the face of it. The details
>> below clarify a lot, as does knowing how the previous discussions have
>> gone. But it would help a lot if the introduction to this PEP were
>> clearer.
>>
>> > The goal of this PEP is get distutils-sig out of the business of being
>> > a gatekeeper for