On Oct 11, 2015 11:07 PM, "Robert Collins" <[email protected]> wrote: > > EWOW, huge thread. > > I've read nearly all of it but in order not to make it massively > worse, I'm going to reply to all the points I think need raising in > one mail :). > > Top level thoughts here, more point fashion with only rough editing > below the fold. > > I realise many things - like the issue between different wheels of the > same package consuming different numpy abis - have been touched on, > but AFAICT they are entirely orthogonal to the proposal, which was to > solve 'be able to use arbitrary build systems and still install with > pip'. > > Of the actual problems with using arbitrary build systems, 99% of them > seem to boil down to 'setup-requires isn't introspectable by pip > (https://github.com/pypa/pip/issues/1820 ). - If it was, then > alternative build systems could be depended on reasonably; and the > mooted thunk from setuptools CLI to arbitrary build system would be > viable. > > It is, in principle a matter of one patch to teach pip *a* way to do > this (and then any and all build systems that want to can utilise it). > https://github.com/rbtcollins/pip/tree/declarative is a POC I did - my > next steps on that were to discuss the right ecosystem stuff for it - > e.g. should pip consume it via setuptools, or should pip support it as > *the way* and other systems including setuptools can choose to use it?
as a standard RDF graph representation, JSON-LD would be uniquely portable here. "PEP 426: Define a JSON-LD context as part of the proposal" https://github.com/pypa/interoperability-peps/issues/31 > > A related but separate thing is being able to *exclusively* install > things without setuptools present - I've filed > https://github.com/pypa/pip/issues/3175 about that, but I think its > -much- lower priority than reliably enabling third party build tools. peep may not need setuptools? * SHA256 * --no-deps * https://pypi.python.org/pypi/peep * wheels > > -Rob > > ---- > > > " > solved many of the hard problems here -- e.g. it's no longer necessary > that a build system also know about every possible installation > configuration -- so pretty much all we really need from a build system > is that it have some way to spit out standard-compliant wheels. > " > > Actually pip still punts a *lot* here - we have bypasses to let things > like C compiler flags be set during wheel build, and when thats done > we don't cache the wheels (or even try to build wheels). > > " > While ``distutils`` / ``setuptools`` have taken us a long way, they > suffer from three serious problems: ... > (c) you are forced to use them anyway, because they provide the > standard interface for installing python packages expected by both > users and installation tools like ``pip``." > > I don't understand the claim of (c) here - its entirely possible to > write a package that doesn't use setuptools and have it do the right > thing - pip uses a subprocess to drive package installation, and the > interface is documented. The interface might be fugly as, but it > exists and works. It is missing setup-requires handling, but so is > setup.py itself. The only thing we'd really need to do AFAICT is make > our setuptools monkeypatching thunk handle setuptools not being > installed (which would be a sensible thing to Just Do anyhow). > > " > - query for build dependencies > - run a build, producing wheels as output > - set up the current source tree so that it can be placed on > ``sys.path`` in "develop mode" > " > > So we have that already. setup.py egg-info, setup.py bdist_wheel, > setup.py develop. > > "A version 1-or-greater format source tree can be identified by the > presence of a file ``_pypackage/_pypackage.cfg``. > " > > I really don't like this. Its going to be with us forever, and its > intrusive (its visible), and so far isn't shown to be fixing anything. > > > "to scatter files around willy-nilly never works, so we adopt the > convention that names starting with an underscore are reserved for > official use, and non-underscored names are available for > idiosyncratic use by individual projects." > > I can see the motivation here, but is it really solving a problem we have? > > > On the specifics of the format: I don't want to kibbitz over strawman > aspects at this point. > > Having the extension mechanism be both pip specific and in Python > means that we're going to face significant adoption issues: the former > because pip is not by any means the only thing around - and some > distros have until very recently been actively hostile to pip (which > in turn means we need to wait a decade or two for them to age-out and > stop being used). The latter because we'll face all the headaches of > running arbitrary untrusted code and dealing with two deps with > different versions of the same hook and so on: I think its an > intrinsically unsafe design. > > @dstufft "problem with numpy.distutils, as I know you’re aware!). We > could do a minimal extension and add another defacto-ish standard of > allowing pip and setuptools to process additional setup_requires like > arguments from a setup.cfg to solve that problem though. The flip side > to this is that since it involves new capabilities in > pip/setuptools/any other installer is that it you’ll have several > years until you can depend on setup.cfg based setup_requires from > being able to be depended on. > " > > Well. For *any* proposal that involves modifying pip, we have to > assume that all existing things keep working, and that anyone wanting > to utilise the new thing will have to either a) include a local > compatibility thunk, or b) error when being used from a too-old > toolchain. I don't think that should really be a factor in design > since its intrinsic to the quagmire. > > "Longer term, I think the answer is sdist 2.0 which has proper > metadata inside of it (name, version, dependencies, etc) but which > also includes a hook like this PEP has to specify the build system > that should be used to build a wheel out of this source distribution." a composed JSON-LD document indicating provenance (who, what, when) for each part of the build chain [VCS archive, egg-info, sdist, wheel, bdist] pydist.jsonld? > > Any reason that can't just be setup.cfg ? > > @Daniel "I thought Robert Collins had a working setup-requires > implementation already? I have a worse but backwards compatible one > too at https://bitbucket.org/dholth/setup-requires/src/tip/setup.py"; - > https://github.com/rbtcollins/pip/tree/declarative - I'll be updating > that probably early next year at this rate - after issue-988 anyhow. > The issue with your approach is that pip doesn't handle having > concurrent installs done well - and in fact it will end up locking its > environment somehow. > > @Paul " > I can understand that a binary wheel may need a certain set of > libraries installed - but that's about the platform tags that are part > of the wheel definition, not about dependencies. Platform tags are an > ongoing discussion, and a good example of a partial solution that" - > thats where the draft PEP tennessee and I start is aimed - at making > those libraries be metadata, not platform tags. > > @Chris " > A given package might depend on numpy, as you say, and it may work > with all numpy versions 1.6 to 1.9. Fine, so we specify that in > install_requires. And this shodl be the dependency in the sdist, too. > If the package is pur python, this is fine and done. > > But if the package has some extensions code that used the numpy C API > ( a very common occurrence), then when it is built, it will only work > (reliably) with the version of numpy it was built with. > > So the project itself, and the sdist depend on numpy >=1.6, but a > build binary wheel depends on numpy == 1.7 (for instance). > > Which requires a binary (wheel) dependency that is somewhat different > than the source dependency. > " - so yes, that is where bdist_wheel should be creating different > metadata for that wheel. The issue that arises is that we need unique > file names so that they can coexist on PyPI or local archives - which > is where wheel tags come in. I'd be in favour of not using semantic > tags for this - rather hash the deps or something and just make a > unique file name. Use actual metadata for metadata. > > @Nathaniel "I know that one unpleasant aspect of the current design is that the > split between egg-info and actual building creates the possibility for > time-of-definition-to-time-of-use bugs, where the final wheel > hopefully matches what egg-info said it would, but in practice there > could be skew. (Of course this is true in any system which represents" > - actually see https://bugs.launchpad.net/pbr/+bug/1502692 for a bug > where this 'skew' is desirable: for older environments we want > tailored deps with no markers, for anything supporting markers we want > them - so the wheel will have markers and egg_info won't. > > @Nathaniel " > (Part of the intuition for the last part is that we also have a > not-terribly-secret-conspiracy here for writing a PEP to get Linux > wheels onto PyPI and at least achieve feature parity with Windows / OS > X. Obviously there will always be weird platforms -- iOS and FreeBSD > and Linux-without-glibc and ... -- but this should dramatically reduce > the frequency with which people need sdist dependencies.)" - I think a > distinction between sdist and binary names for dependencies would be a > terrible mistake. It will raise complexity for reasoning and > describing things without solving any concrete problem that I can see. > > @Nathaniel "I guess to make progress in this conversation I need some > more detailed explanations. I totally get that there's a long history > of thought and conversations behind the various assertions here like > "a sdist is fundamentally different from a VCS checkout", "there must > be a 1-1 mapping between sdists and wheels", "pip needs sdists that > have full wheel metadata in static form", and I'm barging in from the > outside with no context, but I literally have no idea why the specific > design features you're asking for are desirable or even viable. Right > now if I were to try and write the PEP you're asking for, then the > rationale section would just be "because Donald said so" over and over > :-). I couldn't write the motivation section, because I don't know any > problems that the PEP you're describing would fix for me as a package > author (which doesn't mean they don't exist, but!)." -- VCS trees are > (generally) by-humans for humans. They are the primary source of data > and can do thinks like inferring versions from commit data. sdists are > derived from the VCS tree and can include extra data (such as > statically defined version data). Wheels are derived from a tree on > disk and can (today) be built from either VCS trees or sdists. I'm not > sure that forcing an sdist step is beneficial - the egg-info step we > have today is basically that without the cost of compressing and > decompressing potentially large trees for no reason. > > @Jeremy "An sdist is an installable package which just happens to _look_ a > lot like a source release tarball, but trying to pretend that > downstream packagers will want to use it as such leads to a variety > of pain points in the upstream/downstream relationship. For better > or worse a lot of distros don't want generated files in upstream > source code releases, since they need to confirm that they also ship > the necessary tooling to regenerate any required files and that the > generated files they ship match what their packaged tooling > produces." - Well, pbr doesn't work if you just tar up or git export > your VCS tree: it requires the chance to add metadata. And while > distros have whinged about pbr in a number of contexts, that hasn't > been one so far. Downstreams are pretty used to receiving tarballs > with generated files in them - as long as they *have the option* to > recreate those, so the source material isn't lost. [And for version > data, 'grab from git' is a valid answer there']. OTOH perhaps > ftpmaster just hasn't noticed and we're about to get a bug report ;) another interesting use case for [not-] pip: https://github.com/mitsuhiko/pipsi > _______________________________________________ > Distutils-SIG maillist - [email protected] > https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - [email protected] https://mail.python.org/mailman/listinfo/distutils-sig
