Re: [Distutils] PEP: Build system abstraction for pip/conda etc
On Wed, Feb 10, 2016 at 2:43 PM, Paul Moore wrote: > On 10 February 2016 at 13:23, Nick Coghlan wrote: > > On 10 February 2016 at 20:53, Paul Moore wrote: > >> We don't have to solve the whole "sdist 2.0" issue right now. Simply > >> saying that in order to publish pypa.json-based source trees you need > >> to zip up the source directory, name the file "project-version.zip" > >> and upload to PyPI, would be sufficient as a short-term answer > >> (assuming that this *would* be a viable "source file" that pip could > >> use - and I must be clear that I *haven't checked this*!!!) > This is exactly what pip itself does right now for "pip install .", so clearly it is viable. until > >> something like Nathaniel's source distribution proposal, or a > >> full-blown sdist-2.0 spec, is available. We'd need to support whatever > >> stopgap proposal we recommend for backward compatibility in those new > >> proposals, but that's a necessary cost of not wanting to delay the > >> current PEP on those other ones. > > > > One of the reasons I went ahead and created the specifications page at > > https://packaging.python.org/en/latest/specifications/ was to let us > > tweak interoperability requirements as needed, without wasting > > people's time with excessive PEP wrangling by requiring a separate PEP > > for each interface affected by a proposal. > > > > In this case, the build system abstraction PEP should propose some > > additional text for > > > https://packaging.python.org/en/latest/specifications/#source-distribution-format > > defining how to publish source archives containing a pypa.json file > > and the setup.py shim. > The setup.py shim should be optional right? If a package author decides to not care about older pip versions, then the shim isn't needed. Ralf ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP: Build system abstraction for pip/conda etc
On Wed, Feb 10, 2016 at 3:30 PM, David Cournapeau wrote: > > > > On Wed, Feb 10, 2016 at 1:52 PM, Paul Moore wrote: > >> We should probably also check with the flit people that the proposed >> approach works for them. (Are there any other alternative build >> systems apart from flit that exist at present?) >> > > I am not working on it ATM, but bento was fairly complete and could > interoperate w/ pip (a few years ago at least): > https://cournape.github.io/Bento/ > I plan to test with Bento (I'm still using it almost daily to work on Scipy) when an implementation is proposed for pip. The interface in the PEP is straightforward though, I don't see any fundamental reason why it wouldn't work for Bento if it works for flit. Ralf ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 11 February 2016 at 13:48, Nick Coghlan wrote: > On 11 February 2016 at 08:12, Barry Warsaw wrote: >> It's not impossible to migrate to something else, but it's impractical to >> migrate to dozens of something elses. Right now, if we can count on PyPI >> having the source in an easily consumable lowest common denominator format, >> the friction of providing those packages to *our* end users, and updating >> them >> in a timely manner, is often minimal. Changing that ecosystem upstream of >> us, >> either deliberately or otherwise, will likely result in more out of date >> packages in the distros. > > One of my own overarching goals in all this is to help facilitate > utilities like pyp2rpm and py2dsc Hmm, I got the py2dsc reference from https://wiki.debian.org/Python/Packaging but the newer https://wiki.debian.org/Python/LibraryStyleGuide doesn't appear to mention any particular way of generating the initial packaging skeleton from the upstream project. Anyway, the core point is wanting to ensure we can automate not only "direct to binary" installation with Python specific tools, but also the "convert to alternate source archive format and build from there" workflows needed by redistributor ecosystems like Linux distros, conda, Canopy, PyPM, Nix, etc. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 11 February 2016 at 08:12, Barry Warsaw wrote: > It's not impossible to migrate to something else, but it's impractical to > migrate to dozens of something elses. Right now, if we can count on PyPI > having the source in an easily consumable lowest common denominator format, > the friction of providing those packages to *our* end users, and updating them > in a timely manner, is often minimal. Changing that ecosystem upstream of us, > either deliberately or otherwise, will likely result in more out of date > packages in the distros. One of my own overarching goals in all this is to help facilitate utilities like pyp2rpm and py2dsc being able to produce policy compliant distro packages from upstream Python projects *without* any manual fiddling (at least in the case of pure Python modules and packages, and hopefully eventually for extension modules as well), so I'm definitely keeping an eye on the "easy, reliable and automatable access to source code" aspect. Maven (and Maven Central) don't actively encourage machine readable access to source code for published binary artifacts, which turns out to be one of the key problems that makes integrating the JVM ecosystem into Linux distributions a bit of a horror show. Improvements in Linux container tech help with that by putting up a clear "Somebody Else's Problem" field at the container boundary, but it's still a workaround for the historical lack of effective collaboration between the two communities, rather than a real solution. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On Feb 10, 2016, at 10:08 AM, Paul Moore wrote: >But those people will then find that distributing their sources isn't >something that flit covers, so they'll make up their own approach (if it were >me, I'd probably just point people at the project's github account). > >Once people get set up with a workflow that goes like this (build >wheels and point people to github for source) it'll be harder to >encourage them later to switch *back* to a process of uploading >sources to PyPI. > >And that I do think is bad - that we end up pushing people who would >otherwise happily use PyPI for source and binary hosting, to end up >with a solution where they host binaries only on PyPI and make the >source available via another (non-standardised) means. That worries me a lot. Think of the downstream consumers who aren't end users, e.g. Linux distro developers. Some distros have strict requirements on the availability of the source, reproducibility of builds, and so on, along with stacks of tooling that are built on downloading tarballs from PyPI. It's not impossible to migrate to something else, but it's impractical to migrate to dozens of something elses. Right now, if we can count on PyPI having the source in an easily consumable lowest common denominator format, the friction of providing those packages to *our* end users, and updating them in a timely manner, is often minimal. Changing that ecosystem upstream of us, either deliberately or otherwise, will likely result in more out of date packages in the distros. Cheers, -Barry pgpI7uxU9PrYr.pgp Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions
On 10.02.2016 21:26, Donald Stufft wrote: On Feb 10, 2016, at 3:18 PM, Matthias Klose wrote: But then why call it manylinux instead of centos5? You build it on this OS, you expect others to build it on this OS. just name it what it is. Because this is a very specific subset of CentOS 5 that has shown to, in practice, work cross distro into the vast majority of glibc using Linux distributions. The idea here is that if you restrict yourself to these subset of libraries, you’ll produce a wheel that can be installed on say, Debian (assuming a recent enough Debian is used, which is why we’re using the very ancient CentOS 5, so it’s sufficiently old) and not have ABI issues to contend with. “manylinux” is a nicer name than “builtoncentos5butinawaythatisusableonmanyotherlinuxsystems”. the python community has a "good" history calling things if they don't like them. the most longest option names were invented by the setuptools maintainers. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions
> On Feb 10, 2016, at 3:18 PM, Matthias Klose wrote: > > But then why call it manylinux instead of centos5? You build it on this OS, > you expect others to build it on this OS. just name it what it is. Because this is a very specific subset of CentOS 5 that has shown to, in practice, work cross distro into the vast majority of glibc using Linux distributions. The idea here is that if you restrict yourself to these subset of libraries, you’ll produce a wheel that can be installed on say, Debian (assuming a recent enough Debian is used, which is why we’re using the very ancient CentOS 5, so it’s sufficiently old) and not have ABI issues to contend with. “manylinux” is a nicer name than “builtoncentos5butinawaythatisusableonmanyotherlinuxsystems”. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions
On 02.02.2016 01:30, Donald Stufft wrote:
On Feb 1, 2016, at 6:37 PM, Matthias Klose wrote:
On 30.01.2016 00:29, Nathaniel Smith wrote:
Hi all,
I think this is ready for pronouncement now -- thanks to everyone for
all their feedback over the last few weeks!
I don't think so. I am biased because I'm the maintainer for Python in
Debian/Ubuntu. So I would like to have some feedback from maintainers of
Python in other Linux distributions (Nick, no, you're not one of these).
The proposal just takes some environment and declares that as a standard. So
everybody wanting to supply these wheels basically has to use this environment.
Without giving any details, without giving any advise how to produce such
wheels in other environments. Without giving any hints how such wheels may be
broken with newer environments.
I’m not sure this is true. It tells you exactly what versions of glibc and
other libraries it is allowed to link against. It can link against older if it
wants, it can’t link against newer.
Without mentioning this is am64/i386 only.
First sentence: This PEP proposes the creation of a new platform tag for Python
package built distributions, such as wheels, calledmanylinux1_{x86_64,i686}
with external dependencies limited to a standardized, restricted subset of the
Linux kernel and core userspace ABI.
I read "such as wheels, calledmanylinux1_{x86_64,i686}" as not limited to such
platforms.
Later on: Because CentOS 5 is only available for x86_64 and i686 architectures,
these are the only architectures currently supported by the manylinux1 policy.
sorry, didn't see that.
I think it’s a reasonable policy too, AMD64 is responsible for an order of
magnitude more downloads than all other architectures on Linux combined
(71,424,040 vs 1,086,527 in current data set). If you compare AMD64+i386
against everything else then you’re looking at two orders of magnitude
(72,142,511 vs 368,056). I think we can live with a solution that covers 99.5%
of all Linux downloads from PyPI.
But then why call it manylinux instead of centos5? You build it on this OS, you
expect others to build it on this OS. just name it what it is.
There might be more. Pretty please be specific about your environment. Have a
look how the LSB specifies requirements on the runtime environment ... and then
ask yourself why the lsb doesn't have any real value.
Instead of vague references to the LSB, can you tell us why you think the LSB
doesn’t have any real value, and importantly, how that relates to trying to
determine a minimum set of binary ABI. In addition, can you clarify why, if
your assertion is this isn’t going to work, can you state why it won’t work
when it is working for many user’s in the wild through Anaconda, Enthought, the
Holy Build Box, etc?
I'm not seeing the LSB used in real life (anymore), and not any recent updates.
Furthermore, LSB packages were removed in Debian [1].
[1] https://lists.debian.org/debian-lsb/2015/07/msg0.html,
https://lists.debian.org/debian-lsb/2015/07/msg2.html
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 11 February 2016 at 02:36, M.-A. Lemburg wrote: >> Currently what pip does is to >> invoke >> >> $ python setup.py egg_info --egg-base $TEMPDIR >> >> to get the metadata. It is not possible to get the metadata without >> executing the setup.py which is problematic for many applications. >> Providing a static pypa.json file is much better: tools can read a >> static file to get the metadata. > > Depends on which kind of meta data you're after. sdist packages > do include the static PKG-INFO file which has the version 1.0 > meta data. This doesn't include dependencies or namespace > details, but it does have important data such as version, > package name, description, etc. For pip to use it, it needs to include - reliably - version, name, and dependencies; for it to be in any way better, we also need setup-requires or a functional equivalent. Today, we can't rely on the PKG-INFO being complete, so we assume they are all wrong and start over. One of the things we'll get by being strict in subsequent iterations is the ability to rely on things. > In the end, you'll just be defining a different standard > to express the same thing in different ways. > > The setup.py interface was never designed with integration in mind > (only with the idea to provide an extensible interface; and I'm not > going get into the merits of setuptools additions such as > --single-version-externally-managed :-)) but it's still > quite usable for the intended purpose. However we *are defining an integration point*, which is yet another reason not to use the setup.py interface. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 11 February 2016 at 01:21, M.-A. Lemburg wrote: > On 10.02.2016 12:10, Paul Moore wrote: >> On 10 February 2016 at 10:23, M.-A. Lemburg wrote: >>> IMO, that's easy to achieve, though, with the existing de-facto >>> standard interface we already have: the setup.py command line API. >>> We'd just need to publish the minimal set of commands and options, >>> installer will want to see implemented in order to initiate >>> the builds. >> >> No-one who's investing time in writing PEPs is willing to thrash out >> the details of how to use the setup.py interface in a formal proposal >> that sticks to the sort of "minimum required" spec that alternative >> tools would be willing to support. And there's no indication that tool >> developers are willing to implement a setup.py compatible interface >> format as you suggest. And finally, you'd need a way to declare that >> pip installs tool X before trying to run setup.py. > > I don't think that installing 3rd party tools is within the scope > of such a proposal. The setup.py of packages using such tools would > have to either define a dependency to have the installer get the > extra tool, download and install it directly when needed, or tell > the user how to install the tool. > > Alternatively, the package distro could simply ship the tool > embedded in the package. That's what we're doing with > mxSetup.py. > >> So "easy to achieve" still needs someone to take the time to deal with >> these sorts of issue. It's the usual process of the people willing to >> put in the effort get to choose the direction (which is also why I >> just provide feedback, and don't tend to offer my own proposals, >> because I'm not able to commit that sort of time). > > Wait. You are missing the point that the setup.py interface > already does work, so no extra effort is needed. All that's > needed is some documentation of what's currently being used, > so that other tools can support the interface going forward. > > At the moment, pip this interface is only defined by > "what pip uses" and that's a moving target. I disagree with the claim that setup.py already works. Right now the vast majority of setup.py's will end up invoking easy-install, which has entirely separate configuration to pip for networking. It also means that pip is utterly incapable of caching and reusing setup_requires dependencies, so when e.g. numpy is a setup-requires dependency, build times can be arbitrarily high as numpy gets built 3, 4, 5 or more times. Secondly, the setup.py interface includes 'install' as a verb, and there is pretty solid consensus that that is a bug - any definition of 'what pip uses' has to include that as a fallback, for the same reasons pip has to fallback to that - but there is a substantial difference between the end user UX setuptools provides, the interface pip is forced to use, and the smaller one pip would /like/ to use. So all the variants of the PEP we've been discussing are about a modest step forward, capturing pip's existing needs and addressing the setup-requires/easy-install bug as well as the use of install bug. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP: Build system abstraction for pip/conda etc
On Wed, Feb 10, 2016 at 1:52 PM, Paul Moore wrote: > On 10 February 2016 at 13:43, Paul Moore wrote: > >> In this case, the build system abstraction PEP should propose some > >> additional text for > >> > https://packaging.python.org/en/latest/specifications/#source-distribution-format > >> defining how to publish source archives containing a pypa.json file > >> and the setup.py shim. At that point, it will effectively become the > >> spec for sdist 1.0, since that's never previously been officially > >> defined. > >> > >> The key difference from setuptools is that the setup.py shim will be a > >> standard one that flit (and other source archive creation tools) can > >> inject when building the sdist, rather than needing to be a custom > >> file stored in each project's source repository. > > > > Neat! That sounds like a sensible approach, and if the build system > > abstraction PEP adds this, then that addresses my remaining > > objections. > > We should probably also check with the flit people that the proposed > approach works for them. (Are there any other alternative build > systems apart from flit that exist at present?) > I am not working on it ATM, but bento was fairly complete and could interoperate w/ pip (a few years ago at least): https://cournape.github.io/Bento/ > > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP: Build system abstraction for pip/conda etc
On 10 February 2016 at 13:43, Paul Moore wrote: >> In this case, the build system abstraction PEP should propose some >> additional text for >> https://packaging.python.org/en/latest/specifications/#source-distribution-format >> defining how to publish source archives containing a pypa.json file >> and the setup.py shim. At that point, it will effectively become the >> spec for sdist 1.0, since that's never previously been officially >> defined. >> >> The key difference from setuptools is that the setup.py shim will be a >> standard one that flit (and other source archive creation tools) can >> inject when building the sdist, rather than needing to be a custom >> file stored in each project's source repository. > > Neat! That sounds like a sensible approach, and if the build system > abstraction PEP adds this, then that addresses my remaining > objections. We should probably also check with the flit people that the proposed approach works for them. (Are there any other alternative build systems apart from flit that exist at present?) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
Let me speak up about a different and pressing problem: the problem of source code that is not distributed with a GNU automake script. First, any alleged "software" that doesn't use GNU automake is not real and/or should be considered closed source. Second, automake is the best build system that I can imagine. Third, WeirdHat Linux does not understand how to package software that uses CMake. Q.E.D. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP: Build system abstraction for pip/conda etc
On 10 February 2016 at 13:23, Nick Coghlan wrote: > On 10 February 2016 at 20:53, Paul Moore wrote: >> We don't have to solve the whole "sdist 2.0" issue right now. Simply >> saying that in order to publish pypa.json-based source trees you need >> to zip up the source directory, name the file "project-version.zip" >> and upload to PyPI, would be sufficient as a short-term answer >> (assuming that this *would* be a viable "source file" that pip could >> use - and I must be clear that I *haven't checked this*!!!) until >> something like Nathaniel's source distribution proposal, or a >> full-blown sdist-2.0 spec, is available. We'd need to support whatever >> stopgap proposal we recommend for backward compatibility in those new >> proposals, but that's a necessary cost of not wanting to delay the >> current PEP on those other ones. > > One of the reasons I went ahead and created the specifications page at > https://packaging.python.org/en/latest/specifications/ was to let us > tweak interoperability requirements as needed, without wasting > people's time with excessive PEP wrangling by requiring a separate PEP > for each interface affected by a proposal. > > In this case, the build system abstraction PEP should propose some > additional text for > https://packaging.python.org/en/latest/specifications/#source-distribution-format > defining how to publish source archives containing a pypa.json file > and the setup.py shim. At that point, it will effectively become the > spec for sdist 1.0, since that's never previously been officially > defined. > > The key difference from setuptools is that the setup.py shim will be a > standard one that flit (and other source archive creation tools) can > inject when building the sdist, rather than needing to be a custom > file stored in each project's source repository. Neat! That sounds like a sensible approach, and if the build system abstraction PEP adds this, then that addresses my remaining objections. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 10.02.2016 14:00, Oscar Benjamin wrote: > On 10 February 2016 at 12:21, M.-A. Lemburg wrote: >>> So "easy to achieve" still needs someone to take the time to deal with >>> these sorts of issue. It's the usual process of the people willing to >>> put in the effort get to choose the direction (which is also why I >>> just provide feedback, and don't tend to offer my own proposals, >>> because I'm not able to commit that sort of time). >> >> Wait. You are missing the point that the setup.py interface >> already does work, so no extra effort is needed. All that's >> needed is some documentation of what's currently being used, >> so that other tools can support the interface going forward. > > You can see an example of a minimal setup.py file here: > > https://github.com/oscarbenjamin/setuppytest/blob/master/setuppytest/setup.py > > I wrote that some time ago and don't know if it still works (that's > the problem with just having a de facto standard). Agreed, and something that we should address in a PEP. >> At the moment, pip this interface is only defined by >> "what pip uses" and that's a moving target. > > The setup.py interface is a terrible interface for tools like pip to > use and for tools like flit to emulate. I'm not saying that it's a great interface, but it's one that by far most sdists out there support. > Currently what pip does is to > invoke > > $ python setup.py egg_info --egg-base $TEMPDIR > > to get the metadata. It is not possible to get the metadata without > executing the setup.py which is problematic for many applications. > Providing a static pypa.json file is much better: tools can read a > static file to get the metadata. Depends on which kind of meta data you're after. sdist packages do include the static PKG-INFO file which has the version 1.0 meta data. This doesn't include dependencies or namespace details, but it does have important data such as version, package name, description, etc. > To install a distribution pip runs: > > $ python setup.py install --record $RECORD_FILE \ > --single-version-externally-managed > > So the setup.py is entirely responsible not just for building but also > for installing everything. This makes it very difficult to develop a > system where different installer tools and different build tools can > cooperate to allow end users to specify installation options. It also > means that the installer has no direct control over where any of the > files are installed. Why is that ? The install command is very flexible in allowing you to define where the various parts are installed. When defining a minimal set of supported options, the various --install-* options should be part of this. It would also be possible to separate the build and install steps, since distutils is well capable to do this. However, I'm not sure where this aspect fits in relation to the proposed PEP, since it is targeting the operation of building the package and wrapping it into a wheel file, so the bdist_wheel command would have to be used instead. pip wheel pkg runs this command: python setup.py bdist_wheel -d targetdir > If you were designing this from scratch then there are some obvious > things that you would want to do differently here. The setup.py > interface also has so much legacy usage that it's difficult for > setuptools and pip to evolve. The idea with this proposal is to > decouple things by introducing a new interface with well defined and > sensible behaviour. In the end, you'll just be defining a different standard to express the same thing in different ways. The setup.py interface was never designed with integration in mind (only with the idea to provide an extensible interface; and I'm not going get into the merits of setuptools additions such as --single-version-externally-managed :-)) but it's still quite usable for the intended purpose. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 10 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ 2016-01-19: Released eGenix pyOpenSSL 0.13.13 ... http://egenix.com/go86 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP: Build system abstraction for pip/conda etc
On 10 February 2016 at 20:53, Paul Moore wrote: > We don't have to solve the whole "sdist 2.0" issue right now. Simply > saying that in order to publish pypa.json-based source trees you need > to zip up the source directory, name the file "project-version.zip" > and upload to PyPI, would be sufficient as a short-term answer > (assuming that this *would* be a viable "source file" that pip could > use - and I must be clear that I *haven't checked this*!!!) until > something like Nathaniel's source distribution proposal, or a > full-blown sdist-2.0 spec, is available. We'd need to support whatever > stopgap proposal we recommend for backward compatibility in those new > proposals, but that's a necessary cost of not wanting to delay the > current PEP on those other ones. One of the reasons I went ahead and created the specifications page at https://packaging.python.org/en/latest/specifications/ was to let us tweak interoperability requirements as needed, without wasting people's time with excessive PEP wrangling by requiring a separate PEP for each interface affected by a proposal. In this case, the build system abstraction PEP should propose some additional text for https://packaging.python.org/en/latest/specifications/#source-distribution-format defining how to publish source archives containing a pypa.json file and the setup.py shim. At that point, it will effectively become the spec for sdist 1.0, since that's never previously been officially defined. The key difference from setuptools is that the setup.py shim will be a standard one that flit (and other source archive creation tools) can inject when building the sdist, rather than needing to be a custom file stored in each project's source repository. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 10 February 2016 at 22:21, M.-A. Lemburg wrote: > Wait. You are missing the point that the setup.py interface > already does work, so no extra effort is needed. All that's > needed is some documentation of what's currently being used, > so that other tools can support the interface going forward. One of the key points of the proposal is to be able to write *one* setuptools/distutils shim, and then never having to write another one, regardless of how many build systems people come up with [1]. The build system abstraction PEP itself comes from figuring out what pip needs (i.e. the "minimal interface" you're after), and documenting that specifically, without the distracting noise that comes from documenting it in terms of "how pip calls setup.py" (which includes things like passing "--single-version-externally-managed", which only makes sense in the context of setuptools originally being designed to serve the needs of the Chandler project). Cheers, Nick. [1] If we never end up with a build system called "rennet", I am going to be most disappointed :) -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 10 February 2016 at 12:21, M.-A. Lemburg wrote: >> So "easy to achieve" still needs someone to take the time to deal with >> these sorts of issue. It's the usual process of the people willing to >> put in the effort get to choose the direction (which is also why I >> just provide feedback, and don't tend to offer my own proposals, >> because I'm not able to commit that sort of time). > > Wait. You are missing the point that the setup.py interface > already does work, so no extra effort is needed. All that's > needed is some documentation of what's currently being used, > so that other tools can support the interface going forward. You can see an example of a minimal setup.py file here: https://github.com/oscarbenjamin/setuppytest/blob/master/setuppytest/setup.py I wrote that some time ago and don't know if it still works (that's the problem with just having a de facto standard). > At the moment, pip this interface is only defined by > "what pip uses" and that's a moving target. The setup.py interface is a terrible interface for tools like pip to use and for tools like flit to emulate. Currently what pip does is to invoke $ python setup.py egg_info --egg-base $TEMPDIR to get the metadata. It is not possible to get the metadata without executing the setup.py which is problematic for many applications. Providing a static pypa.json file is much better: tools can read a static file to get the metadata. To install a distribution pip runs: $ python setup.py install --record $RECORD_FILE \ --single-version-externally-managed So the setup.py is entirely responsible not just for building but also for installing everything. This makes it very difficult to develop a system where different installer tools and different build tools can cooperate to allow end users to specify installation options. It also means that the installer has no direct control over where any of the files are installed. If you were designing this from scratch then there are some obvious things that you would want to do differently here. The setup.py interface also has so much legacy usage that it's difficult for setuptools and pip to evolve. The idea with this proposal is to decouple things by introducing a new interface with well defined and sensible behaviour. -- Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 10.02.2016 12:10, Paul Moore wrote: > On 10 February 2016 at 10:23, M.-A. Lemburg wrote: >> IMO, that's easy to achieve, though, with the existing de-facto >> standard interface we already have: the setup.py command line API. >> We'd just need to publish the minimal set of commands and options, >> installer will want to see implemented in order to initiate >> the builds. > > No-one who's investing time in writing PEPs is willing to thrash out > the details of how to use the setup.py interface in a formal proposal > that sticks to the sort of "minimum required" spec that alternative > tools would be willing to support. And there's no indication that tool > developers are willing to implement a setup.py compatible interface > format as you suggest. And finally, you'd need a way to declare that > pip installs tool X before trying to run setup.py. I don't think that installing 3rd party tools is within the scope of such a proposal. The setup.py of packages using such tools would have to either define a dependency to have the installer get the extra tool, download and install it directly when needed, or tell the user how to install the tool. Alternatively, the package distro could simply ship the tool embedded in the package. That's what we're doing with mxSetup.py. > So "easy to achieve" still needs someone to take the time to deal with > these sorts of issue. It's the usual process of the people willing to > put in the effort get to choose the direction (which is also why I > just provide feedback, and don't tend to offer my own proposals, > because I'm not able to commit that sort of time). Wait. You are missing the point that the setup.py interface already does work, so no extra effort is needed. All that's needed is some documentation of what's currently being used, so that other tools can support the interface going forward. At the moment, pip this interface is only defined by "what pip uses" and that's a moving target. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 10 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ 2016-01-19: Released eGenix pyOpenSSL 0.13.13 ... http://egenix.com/go86 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 10 February 2016 at 10:23, M.-A. Lemburg wrote: > IMO, that's easy to achieve, though, with the existing de-facto > standard interface we already have: the setup.py command line API. > We'd just need to publish the minimal set of commands and options, > installer will want to see implemented in order to initiate > the builds. No-one who's investing time in writing PEPs is willing to thrash out the details of how to use the setup.py interface in a formal proposal that sticks to the sort of "minimum required" spec that alternative tools would be willing to support. And there's no indication that tool developers are willing to implement a setup.py compatible interface format as you suggest. And finally, you'd need a way to declare that pip installs tool X before trying to run setup.py. So "easy to achieve" still needs someone to take the time to deal with these sorts of issue. It's the usual process of the people willing to put in the effort get to choose the direction (which is also why I just provide feedback, and don't tend to offer my own proposals, because I'm not able to commit that sort of time). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP: Build system abstraction for pip/conda etc
On 10 February 2016 at 01:19, Robert Collins wrote: > On 10 February 2016 at 13:09, Paul Moore wrote: >> [I need to read and digest the rest of this, but it's late here, so >> that will be tomorrow] > > OK, cool. Right, I've been thinking about this a bit more, and I've re-read the PEP. I wasn't at all clear in my concern - in my own mind, and as a consequence in how I explained the issue. My apologies. I don't have any problem with the basic proposed interface, and I see what you mean that it doesn't need to require a "sdist" command. I was wrong on that. But the proposal is written in terms of how pip "works from a Python source tree". The example used throughout the PEP is a local source directory. While that's an easy and understandable example to use, it glosses over the detail of how pip will *get* such a source tree. What the PEP omits, it seems to me, is an explanation of how pip will get a "source tree" in the form defined by the PEP. In essence, there is no transition plan in the PEP. At present, pip sets up a source tree by downloading a sdist from PyPI and unpacking it. The PEP explains that sources without a pypa.json should be treated as current setuptools-style sdists for backward compatibility, but offers no guidance on how projects (or build tools, on behalf of projects that use them) can transition to a pypa.json-based approach. That's really what I'm trying to get at with my focus on "source distributions". As a project author, how do I switch to this new format? What do I publish? Publishing binaries in the form of wheels is irrelevant - projects using flit or other build tools can do that right now. This PEP is entirely about enabling them to publish *source* using new build tools, and yet it doesn't cover the publishing side at all. So while the PEP is fine, it's incomplete from the user's point of view, and my concern is that if we finalise it in its current form, people will have to roll their own publishing solutions, and we'll then be playing "catch up" trying to define a standard that takes into account the approaches people have developed in the meantime. And as a user, I'll then be left with projects I can't easily build wheels for. It's all very well to say "the project should build wheels", and I wish they would, but not all do or will - I care because one of my ongoing side projects is an automated wheel build farm (for simple cases only, don't anyone get their hopes up! ;-)) and being able to build wheels in a standard way is key to that. We don't have to solve the whole "sdist 2.0" issue right now. Simply saying that in order to publish pypa.json-based source trees you need to zip up the source directory, name the file "project-version.zip" and upload to PyPI, would be sufficient as a short-term answer (assuming that this *would* be a viable "source file" that pip could use - and I must be clear that I *haven't checked this*!!!) until something like Nathaniel's source distribution proposal, or a full-blown sdist-2.0 spec, is available. We'd need to support whatever stopgap proposal we recommend for backward compatibility in those new proposals, but that's a necessary cost of not wanting to delay the current PEP on those other ones. Hope this is clearer. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 10.02.2016 11:08, Paul Moore wrote: > On 10 February 2016 at 09:34, M.-A. Lemburg wrote: >> I'm not sure I'm parsing your comment correctly, but if you are >> suggesting that PyPI should no longer allow supporting >> non-open-source packages, this is definitely not going to >> happen. > > Not at all. That's good to know :-) > Although as far as I know the number of closed-source packages > on PyPI is vanishingly small... > > My concern is that we seem to be opening up the option of using > non-setuptools build systems without having a good solution for people > *wishing* to upload sources. It's more a matter of timing - if we > allow people to use (say) flit for their builds then presumably a > proportion of people will, because it's easier to use than setuptools, > *for builds*. But those people will then find that distributing their > sources isn't something that flit covers, so they'll make up their own > approach (if it were me, I'd probably just point people at the > project's github account). > > Once people get set up with a workflow that goes like this (build > wheels and point people to github for source) it'll be harder to > encourage them later to switch *back* to a process of uploading > sources to PyPI. > > And that I do think is bad - that we end up pushing people who would > otherwise happily use PyPI for source and binary hosting, to end up > with a solution where they host binaries only on PyPI and make the > source available via another (non-standardised) means. That's a fair argument indeed. > In no way though am I proposing that we stop people making deliberate > choices on how they distribute their packages. Just that we make > hosting both source and binaries on PyPI the "no friction" easy option > for (the majority of?) people who don't really mind, and just want to > make their work publicly available. Well, you know, there's an important difference between making work publicly available and giving away the source code :-) But I get your point and do support it: It should be possible for package authors to use different build tools than distutils. IMO, that's easy to achieve, though, with the existing de-facto standard interface we already have: the setup.py command line API. We'd just need to publish the minimal set of commands and options, installer will want to see implemented in order to initiate the builds. > PS This has gone a long way off the topic of the build interface > proposal, so I'm glad it's been spun off into its own thread. I'm now > of the view that this relates at best peripherally to the build > interface proposal, which I'll comment on in the other thread. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 10 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ 2016-01-19: Released eGenix pyOpenSSL 0.13.13 ... http://egenix.com/go86 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 10 February 2016 at 09:34, M.-A. Lemburg wrote: > I'm not sure I'm parsing your comment correctly, but if you are > suggesting that PyPI should no longer allow supporting > non-open-source packages, this is definitely not going to > happen. Not at all. Although as far as I know the number of closed-source packages on PyPI is vanishingly small... My concern is that we seem to be opening up the option of using non-setuptools build systems without having a good solution for people *wishing* to upload sources. It's more a matter of timing - if we allow people to use (say) flit for their builds then presumably a proportion of people will, because it's easier to use than setuptools, *for builds*. But those people will then find that distributing their sources isn't something that flit covers, so they'll make up their own approach (if it were me, I'd probably just point people at the project's github account). Once people get set up with a workflow that goes like this (build wheels and point people to github for source) it'll be harder to encourage them later to switch *back* to a process of uploading sources to PyPI. And that I do think is bad - that we end up pushing people who would otherwise happily use PyPI for source and binary hosting, to end up with a solution where they host binaries only on PyPI and make the source available via another (non-standardised) means. In no way though am I proposing that we stop people making deliberate choices on how they distribute their packages. Just that we make hosting both source and binaries on PyPI the "no friction" easy option for (the majority of?) people who don't really mind, and just want to make their work publicly available. Paul PS This has gone a long way off the topic of the build interface proposal, so I'm glad it's been spun off into its own thread. I'm now of the view that this relates at best peripherally to the build interface proposal, which I'll comment on in the other thread. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
> Paul Moore writes: > >> But as I said in my response to Nathaniel, it may be that all that is >> needed is some context in the PEP explaining how we require[1] people >> to upload source to PyPI in the new world where we support build >> systems which don't have a "sdist" command like setuptools does. >> >> Paul >> >> [1] I say "require" in the sense of "you have to follow these rules if >> pip is to be able to use your source", not "you must upload source" - >> although I hope that the number of people actually preferring to *not* >> include source in their PyPI uploads is vanishingly small... I'm not sure I'm parsing your comment correctly, but if you are suggesting that PyPI should no longer allow supporting non-open-source packages, this is definitely not going to happen. Python is free for everyone to use without any GPL-like restrictions, which is part of our big success, and our packaging environment has to follow the same principle. The attitude that some people in this discussion are showing does not align with those principles, which I find increasingly worrying. When discussing technicalities in this space, you always have to take the political implications into account as well. Back on topic: I don't think that the build system abstraction is moving in the right direction. Instead of coming up with yet another standard for build interfacing, we should simply pin down the commands and options that pip and other installers will want to see working with the standard setup.py command line interface we have. There aren't all that many - simply take what pip does now as minimal standard. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 10 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ 2016-01-19: Released eGenix pyOpenSSL 0.13.13 ... http://egenix.com/go86 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
