Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 11:46 PM, Donald Stufft wrote: > catalog-sig without an argument? Is this a first? ;) No. This! Is! Spart... uh, I mean, DISTUTILS-SIG! ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 27, 2013, at 11:44 PM, PJ Eby wrote: > On Wed, Mar 27, 2013 at 10:09 PM, Donald Stufft wrote: >> Right, which is why I think the ability to install from a raw source is a >> good feature for an installer, but not for the dependency metadata. > > Sure - I never said the dependency metadata had to be able to say > *where* you get the raw source from, just that the tool for resolving > dependencies needed to be able to process raw source into something > installable. > >> Following that we just need a standard way for a raw source tarball to >> declare what it's builder is, either via some sort of file that tells you >> that, or a build script , or something along those lines. > > Yep. Static configuration is a *must*, here, though, as we want to > move away from arbitrary setup script writing by package authors: in > general they are really bad at it. A lot of setuptools' odd > build-time features (like sandboxing) exist specifically because > people write whatever crap they want in setup.py and have zero idea > how to actually use/integrate with distutils. > > One interesting feature that would be possible under a > configuration-based system is that you could actually have an > installer with a whitelist or blacklist for build tools and > setup-requires, in order to prevent or limit untrusted code execution > by the overall build system. This would make it slightly more > practical to have, say, servers that build wheels, such that only > tools the servers' owners know won't import or run arbitrary code are > allowed to do the compiling. (Not that that should be the only > security involved, but it'd be a cool first-tier sanity check.) > > (Interestingly, this is also an argument for having a separate > "tests-require-dist" in metadata 2.0, since testing tools *have* to > run arbitrary code from the package, but archivers and builders do > not.) catalog-sig without an argument? Is this a first? ;) > > > Nick wrote: >>> However, as I've said elsewhere, for metadata 2.0, I *do not* plan to >>> migrate the archiving or build steps away from setup.py. So "give me >>> an sdist" will be spelled "python setup.py sdist" and "give me a wheel >>> file" will be spelled "python setup.py bdist_wheel". > > Works for me. Well, sort of. In principle, it means you can grow > next generation build systems that use a dummy setup.py. > > In practice, it means you're still gonna be relying on setuptools. > (Presumably 0.7 post-merge, w/bdist_wheel support baked in.) At some > point, there has to be a new way to do it, because the pain of > creating a functional dummy setup.py is a really high barrier to entry > for a build tool to meet, until all the current tools that run > setup.py files go away. > > IMO it'd be better to standardize this bit *now*, so that it'd be > practical to start shipping projects without a setup.py, or perhaps > make a "one dummy setup.py to rule them all" implementation that > delegates everything to the new build interface. > > I can certainly understand that there are more urgent priorities in > the short run; I just hope that a standard for this part lands > concurrent with, say, PEP 439 and distlib ending up in the stdlib, so > we don't have to wait another couple years to begin phasing out > setuptools/distutils as the only build game in town. > > I mean, it basically amounts to defining some parameters to > programmatically call a pair of sdist() and bdist_wheel() functions > with, and a configuration syntax to say what distributions and modules > to import those functions from. So it's not like it's going to be a > huge time drain. (Maybe not even as much has already been consumed by > this thread so far. ;-) ) > > > Nick also wrote: >>> There's also an interesting migration problem for pre-2.0 sdists, >>> where we can't assume that "python setup.py bdist_wheel && pip install >>> " is equivalent to "python setup.py install": projects >>> like Twisted that run a post-install hook won't install properly if >>> you build a wheel first, since the existing post-install hook won't >>> run. >>> >>> It's an interesting problem, but one where my near term plans amount >>> to "document the status quo". > > Yeah, it's already broken and the new world order isn't going to break > it any further. Same goes for allowing pip to convert eggs; the ones > that don't work right due to bad platform tags, etc. *already* don't > work, so documenting the status quo as a transitional measure is > sufficient. Heck, in general, supporting backward compatible stuff > that suffers from the same problems as the stuff it's being backward > compatible with is a no-brainer if it lets people get on the new so we > can phase out the old. > > (Which is why I love that Vinay is looking into how to make wheels > more usable for some of eggs less-frequent but still important use > cases: it makes it that much easier to tell someone they don't need to > stay on setuptools to do the same
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 10:09 PM, Donald Stufft wrote: > Right, which is why I think the ability to install from a raw source is a > good feature for an installer, but not for the dependency metadata. Sure - I never said the dependency metadata had to be able to say *where* you get the raw source from, just that the tool for resolving dependencies needed to be able to process raw source into something installable. > Following that we just need a standard way for a raw source tarball to > declare what it's builder is, either via some sort of file that tells you > that, or a build script , or something along those lines. Yep. Static configuration is a *must*, here, though, as we want to move away from arbitrary setup script writing by package authors: in general they are really bad at it. A lot of setuptools' odd build-time features (like sandboxing) exist specifically because people write whatever crap they want in setup.py and have zero idea how to actually use/integrate with distutils. One interesting feature that would be possible under a configuration-based system is that you could actually have an installer with a whitelist or blacklist for build tools and setup-requires, in order to prevent or limit untrusted code execution by the overall build system. This would make it slightly more practical to have, say, servers that build wheels, such that only tools the servers' owners know won't import or run arbitrary code are allowed to do the compiling. (Not that that should be the only security involved, but it'd be a cool first-tier sanity check.) (Interestingly, this is also an argument for having a separate "tests-require-dist" in metadata 2.0, since testing tools *have* to run arbitrary code from the package, but archivers and builders do not.) Nick wrote: >> However, as I've said elsewhere, for metadata 2.0, I *do not* plan to >> migrate the archiving or build steps away from setup.py. So "give me >> an sdist" will be spelled "python setup.py sdist" and "give me a wheel >> file" will be spelled "python setup.py bdist_wheel". Works for me. Well, sort of. In principle, it means you can grow next generation build systems that use a dummy setup.py. In practice, it means you're still gonna be relying on setuptools. (Presumably 0.7 post-merge, w/bdist_wheel support baked in.) At some point, there has to be a new way to do it, because the pain of creating a functional dummy setup.py is a really high barrier to entry for a build tool to meet, until all the current tools that run setup.py files go away. IMO it'd be better to standardize this bit *now*, so that it'd be practical to start shipping projects without a setup.py, or perhaps make a "one dummy setup.py to rule them all" implementation that delegates everything to the new build interface. I can certainly understand that there are more urgent priorities in the short run; I just hope that a standard for this part lands concurrent with, say, PEP 439 and distlib ending up in the stdlib, so we don't have to wait another couple years to begin phasing out setuptools/distutils as the only build game in town. I mean, it basically amounts to defining some parameters to programmatically call a pair of sdist() and bdist_wheel() functions with, and a configuration syntax to say what distributions and modules to import those functions from. So it's not like it's going to be a huge time drain. (Maybe not even as much has already been consumed by this thread so far. ;-) ) Nick also wrote: >> There's also an interesting migration problem for pre-2.0 sdists, >> where we can't assume that "python setup.py bdist_wheel && pip install >> " is equivalent to "python setup.py install": projects >> like Twisted that run a post-install hook won't install properly if >> you build a wheel first, since the existing post-install hook won't >> run. >> >> It's an interesting problem, but one where my near term plans amount >> to "document the status quo". Yeah, it's already broken and the new world order isn't going to break it any further. Same goes for allowing pip to convert eggs; the ones that don't work right due to bad platform tags, etc. *already* don't work, so documenting the status quo as a transitional measure is sufficient. Heck, in general, supporting backward compatible stuff that suffers from the same problems as the stuff it's being backward compatible with is a no-brainer if it lets people get on the new so we can phase out the old. (Which is why I love that Vinay is looking into how to make wheels more usable for some of eggs less-frequent but still important use cases: it makes it that much easier to tell someone they don't need to stay on setuptools to do the same stuff.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 27, 2013, at 10:05 PM, Nick Coghlan wrote: > On Thu, Mar 28, 2013 at 11:43 AM, Donald Stufft wrote: >> I don't think you can, nor should you be able to, explicitly depend on >> something that is a VCS checkout. > > I find it more useful to think of the issue as whether or not you > allow publication of source tarballs to satisfy a dependency, or > *require* publication of a fully populated sdist. If you allow raw > source tarballs, then you effectively allow VCS checkouts as well. I > prefer requiring an explicit publication step, but we also need to > acknowledge that the installer ecosystem we're trying to replace > allows them, and some people are relying on that feature. Right, which is why I think the ability to install from a raw source is a good feature for an installer, but not for the dependency metadata. Following that we just need a standard way for a raw source tarball to declare what it's builder is, either via some sort of file that tells you that, or a build script , or something along those lines. > > However, as I've said elsewhere, for metadata 2.0, I *do not* plan to > migrate the archiving or build steps away from setup.py. So "give me > an sdist" will be spelled "python setup.py sdist" and "give me a wheel > file" will be spelled "python setup.py bdist_wheel". > > There's also an interesting migration problem for pre-2.0 sdists, > where we can't assume that "python setup.py bdist_wheel && pip install > " is equivalent to "python setup.py install": projects > like Twisted that run a post-install hook won't install properly if > you build a wheel first, since the existing post-install hook won't > run. > > It's an interesting problem, but one where my near term plans amount > to "document the status quo". > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Thu, Mar 28, 2013 at 11:43 AM, Donald Stufft wrote: > I don't think you can, nor should you be able to, explicitly depend on > something that is a VCS checkout. I find it more useful to think of the issue as whether or not you allow publication of source tarballs to satisfy a dependency, or *require* publication of a fully populated sdist. If you allow raw source tarballs, then you effectively allow VCS checkouts as well. I prefer requiring an explicit publication step, but we also need to acknowledge that the installer ecosystem we're trying to replace allows them, and some people are relying on that feature. However, as I've said elsewhere, for metadata 2.0, I *do not* plan to migrate the archiving or build steps away from setup.py. So "give me an sdist" will be spelled "python setup.py sdist" and "give me a wheel file" will be spelled "python setup.py bdist_wheel". There's also an interesting migration problem for pre-2.0 sdists, where we can't assume that "python setup.py bdist_wheel && pip install " is equivalent to "python setup.py install": projects like Twisted that run a post-install hook won't install properly if you build a wheel first, since the existing post-install hook won't run. It's an interesting problem, but one where my near term plans amount to "document the status quo". Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 27, 2013, at 9:03 PM, PJ Eby wrote: > On Wed, Mar 27, 2013 at 6:55 PM, Donald Stufft wrote: >> I still think I prefer a setup.py develop or develop.py to invoke the build >> system for development builds > > It's possible we're not talking about the same thing -- I notice you > keep mentioning "setup.py develop", which is entirely unrelated to the > scenarios I'm talking about. "setup.py develop" is for installing > something *you* are working on/developing. But depending on raw > source doesn't imply that you would be editing or developing that > source; it just means that you have a bleeding-edge dependency (which > might in turn have others), adding to your management overhead if you > have to know how to build an sdist for each of those dependencies > whenever you need a refresh. > > So, I'm not talking about scenarios where a user obtains a source > checkout and does something with it, I'm talking about scenarios where > the developer of a package wants to declare a *dependency* on > *another* package that currently has to be fetched from revision > control. So, in order to install *their* package (e.g. to their > staging/test server), the install system has to be able to fetch and > build from raw sources. I don't think you can, nor should you be able to, explicitly depend on something that is a VCS checkout. Declared dependencies in the metadata should be "abstract", they are a name, possibly a version specifier but they are explicitly _not_ where you get that dependency from. They only become "concrete" when you resolve the abstract dependencies via an index (this index could be PyPI, it could be a directory on your machine, etc). This fits in very well with the idea of "Provides" as well, I do not depend on https://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz I depend on something that claims to be setuptools it could be https://bitbucket.org/tarek/distribute. The point being me as the theoretical author of setuptools package author can't dictate where to install my package from. > > >> Sorry my statement was a bit unclear, those people would all fall under >> actively working on that project (Zope in this case). I mean installs from >> VCS's should be discouraged for end users. > > Define "end users". ;-) > > Here's a different example: there was a point at which I was actively > developing PEAK-Rules and somebody else was actively developing > something that used it. That person wasn't developing PEAK-Rules, and > I wasn't part of their project, but they wanted up-to-the-minute > versions because I was making changes based on their use cases, which > they needed right away. Are they an "end user"? ;-) > > You could argue that, well, that's just one project, except that what > if somebody *else* depends on *their* project, because they're also > doing bleeding edge development? > > Well, that happened, too, because the consumer of PEAK-Rules was doing > a bleeding-edge library that *other* people were doing bleeding-edge > development against. So now there were two levels of dependency on > raw sources. I see, I was misunderstanding the use case. Like I said above though I don't think that a package author should dictate where you install X from. I don't know easy_install or buildout very well, but when I need a bleeding-edge release of something in my dependency graph I use pip's requirements.txt file and add a ``-e <…>`` (which pip internally translates to checking the repo out and running setup.py develop). This is where, in my mind, this belongs because requirements.txt lists "concrete" dependencies (it is paired with a index url, defaulting to PyPI) and so I'm just listing another "concrete" dependency. This does mean dependency graphs involving pre-released unpackaged dependencies are less friendly, but I think that's ok because: * Users should opt into development releases - This thought is reflected in the PEP426 where it instructs installers to default to stable releases only unless the end user requests it either via flag, or by explicitly including it in the version spec. * This is already outside of the packaging infrastructure/ecosystem. Sometimes I need system libraries installed too and I need to manually make sure they are installed. Python packaging can't solve every problem. * I think this is an edge case (one I have hit myself) and I don't think it's a large enough use case to break the "abstract"-ness of the PKG-INFO/JSON/whatever metadata. > > If you don't support these kinds of scenarios, you slow the > community's development velocity. Not too long ago, Richard Jones > posted a graph on r/Python showing how package registration took off > exponentially around the time easy_install was released. I think that > this is in large part due to the increased development velocity > afforded by being able to depend on other packages at both development > *and* deployment time. Even though m
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 6:55 PM, Donald Stufft wrote: > I still think I prefer a setup.py develop or develop.py to invoke the build > system for development builds It's possible we're not talking about the same thing -- I notice you keep mentioning "setup.py develop", which is entirely unrelated to the scenarios I'm talking about. "setup.py develop" is for installing something *you* are working on/developing. But depending on raw source doesn't imply that you would be editing or developing that source; it just means that you have a bleeding-edge dependency (which might in turn have others), adding to your management overhead if you have to know how to build an sdist for each of those dependencies whenever you need a refresh. So, I'm not talking about scenarios where a user obtains a source checkout and does something with it, I'm talking about scenarios where the developer of a package wants to declare a *dependency* on *another* package that currently has to be fetched from revision control. So, in order to install *their* package (e.g. to their staging/test server), the install system has to be able to fetch and build from raw sources. > Sorry my statement was a bit unclear, those people would all fall under > actively working on that project (Zope in this case). I mean installs from > VCS's should be discouraged for end users. Define "end users". ;-) Here's a different example: there was a point at which I was actively developing PEAK-Rules and somebody else was actively developing something that used it. That person wasn't developing PEAK-Rules, and I wasn't part of their project, but they wanted up-to-the-minute versions because I was making changes based on their use cases, which they needed right away. Are they an "end user"? ;-) You could argue that, well, that's just one project, except that what if somebody *else* depends on *their* project, because they're also doing bleeding edge development? Well, that happened, too, because the consumer of PEAK-Rules was doing a bleeding-edge library that *other* people were doing bleeding-edge development against. So now there were two levels of dependency on raw sources. If you don't support these kinds of scenarios, you slow the community's development velocity. Not too long ago, Richard Jones posted a graph on r/Python showing how package registration took off exponentially around the time easy_install was released. I think that this is in large part due to the increased development velocity afforded by being able to depend on other packages at both development *and* deployment time. Even though most packages don't depend on the bleeding edge (because they're not themselves the bleeding edge), for individual development it's a godsend to be able to depend on your *own* packages from revision control, without needing all kinds of manual rigamarole to use them. (This is also really relevant for private and corporate-internal development scenarios.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On 27 March 2013 21:06, Jim Fulton wrote: >> - AFAIK eggs allow you to import extensions >> from zip in a similar fashion. > > Importing from zipped eggs has proved itself to be an > anti pattern. I don't like the idea of making wheels work like eggs in this respect. As Jim said, (zipped) eggs have a very bad reputation and associating wheels with that type of functionality would be a very bad idea. Wheels are, and should remain, a binary installer format. On the other hand, zipimport is a very cool feature, and seriously under-used. But it has specific benefits and limitations, and in particular zipimport does not support binary extensions for very good reasons. Zip files on sys.path are practical for pure Python code only, IMO. Having said all that, the fact that wheels are zipfiles, and can be used on sys.path, *can* be useful. But it's an incidental benefit and *not* a core feature. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 27, 2013, at 6:41 PM, PJ Eby wrote: > On Wed, Mar 27, 2013 at 1:51 PM, Donald Stufft wrote: >> I don't think the packaging formats should dictate the development flow at >> all. .IN files and such all dictate how that should be. > > You can't *not* dictate the flow. If you don't have something to > generate the file with, then you're dictating that developers must > ship an sdist and users must manually run build tools to make wheels. > > (Or, more likely, you're creating the motivation for somebody to > create a meta-meta-build system that solves the problem, probably by > having a bunch of plugins to detect what build system a raw source > directory is using.) > > >> To me this is an installer issue not a packaging issue and it's best solved >> in the installers. Obviously there is some benefit to a "standard" way for >> installers to treat these but I don't think it should be defined in terms of >> the packaging formats. > > It definitely doesn't have to be. distutils2's setup.cfg isn't > actually a bad human-readable format, but it's not a *packaging* > format. > > In any case, the only thing an installer needs is a way to get the > setup-requires-dist, or the portion of it that pertains to identifying > the metadata hooks. The rest could be handled with entry points > registered for configuration file names. For example, Bento could > expose an entry point like: > > [mebs.metadata.generators] > bento.info = some.module.in.bento:hookfunction > > And then an installer builds a list of these hooks that are in the > setup-requires-dists, and runs them based on the filenames found in > the project directory. All done. > > Basically, the only thing we need is a way to avoid having to either: > > 1. Make every user install Bento/etc. before trying to install a > package from source that uses it, or > 2. Embed a registry of every possible build configuration file name > into every installer. > > And this can be done in any of several ways: > > * Have a standardized naming pattern like pybuild.*, and make the .* > part indicate the build tool > > * Have a standardized single name (like pybuild.cfg), and encode the > build tool in a string that can be read regardless of file format, so > it can be embedded in whatever format the build tool itself uses > > * Have a separate file that *only* lists the build tool or > setup-requires-dists (and maybe can be extended to contain other > information for use with a stdlib-supplied build tool) > > I personally lean towards the last one, especially if it reuses > setup.cfg, because setup.cfg already exists and is fairly > standardized. There are even tools that work today to let you do a > metadata-free setup.py and specify everything needed in setup.cfg, > with environment markers and everything. > > Heck, IIUC, there's at least one library you can use today with > *setuptools* to do that -- it doesn't need distutils2 or any of that, > it just translates setup.cfg to setup.py arguments. > > But an even more important reason to standardize is that there should > be one, and preferably only one, obvious way to do it. AFAIK, the > distutils2 effort didn't fail because of setup.cfg -- heck, setup.cfg > was the main *benefit* I saw in the distutils2 work, everything else > about it AFAIK was just setuptools warmed over -- it failed because of > trying to boil the ocean and *implement* everything, rather than just > standardizing on interfaces. A minimal boilerplate setup.cfg could be > something like > > [build] > builder = bento >1.6 > > And leave it at that. Disadvantage is that it's a dumb boilerplate > file for tools that don't use setup.cfg for their configuration -- > i.e., it's a minor disadvantage to users of those tools. However, if > your preferred build tool generates the file for you, it's no big > deal, as long as the generated file doesn't change all the time and > you check it into source control. > > Such a usage pattern is teachable and provides what's needed, without > dictating anything about the development workflow, other than that you > need to tell installers how to make an sdist if you want people to > install stuff you shipped without an sdist or a wheel, or if you want > to use any generic build-running tools that need to know your build > hook(s). repurposing a single line of setup.cfg for this usecase wouldn't be the worst thing in the world. I don't like setup.cfg and I especially don't like it as the format to exchange the _actual_ metadata, but as a configuration format (configure which build system to use) it's ok. I still think I prefer a setup.py develop or develop.py to invoke the build system for development builds, but atm the difference between something like echo "[build]\nbuilder = bento > 1.6" > setup.cfg and develop.py is not a hill I care to die on. Maybe Nick has different ides for how VCS/install from an unpacked directory (E.g. explicitly not a package) should look I don't kn
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 1:51 PM, Donald Stufft wrote: > I don't think the packaging formats should dictate the development flow at > all. .IN files and such all dictate how that should be. You can't *not* dictate the flow. If you don't have something to generate the file with, then you're dictating that developers must ship an sdist and users must manually run build tools to make wheels. (Or, more likely, you're creating the motivation for somebody to create a meta-meta-build system that solves the problem, probably by having a bunch of plugins to detect what build system a raw source directory is using.) > To me this is an installer issue not a packaging issue and it's best solved > in the installers. Obviously there is some benefit to a "standard" way for > installers to treat these but I don't think it should be defined in terms of > the packaging formats. It definitely doesn't have to be. distutils2's setup.cfg isn't actually a bad human-readable format, but it's not a *packaging* format. In any case, the only thing an installer needs is a way to get the setup-requires-dist, or the portion of it that pertains to identifying the metadata hooks. The rest could be handled with entry points registered for configuration file names. For example, Bento could expose an entry point like: [mebs.metadata.generators] bento.info = some.module.in.bento:hookfunction And then an installer builds a list of these hooks that are in the setup-requires-dists, and runs them based on the filenames found in the project directory. All done. Basically, the only thing we need is a way to avoid having to either: 1. Make every user install Bento/etc. before trying to install a package from source that uses it, or 2. Embed a registry of every possible build configuration file name into every installer. And this can be done in any of several ways: * Have a standardized naming pattern like pybuild.*, and make the .* part indicate the build tool * Have a standardized single name (like pybuild.cfg), and encode the build tool in a string that can be read regardless of file format, so it can be embedded in whatever format the build tool itself uses * Have a separate file that *only* lists the build tool or setup-requires-dists (and maybe can be extended to contain other information for use with a stdlib-supplied build tool) I personally lean towards the last one, especially if it reuses setup.cfg, because setup.cfg already exists and is fairly standardized. There are even tools that work today to let you do a metadata-free setup.py and specify everything needed in setup.cfg, with environment markers and everything. Heck, IIUC, there's at least one library you can use today with *setuptools* to do that -- it doesn't need distutils2 or any of that, it just translates setup.cfg to setup.py arguments. But an even more important reason to standardize is that there should be one, and preferably only one, obvious way to do it. AFAIK, the distutils2 effort didn't fail because of setup.cfg -- heck, setup.cfg was the main *benefit* I saw in the distutils2 work, everything else about it AFAIK was just setuptools warmed over -- it failed because of trying to boil the ocean and *implement* everything, rather than just standardizing on interfaces. A minimal boilerplate setup.cfg could be something like [build] builder = bento >1.6 And leave it at that. Disadvantage is that it's a dumb boilerplate file for tools that don't use setup.cfg for their configuration -- i.e., it's a minor disadvantage to users of those tools. However, if your preferred build tool generates the file for you, it's no big deal, as long as the generated file doesn't change all the time and you check it into source control. Such a usage pattern is teachable and provides what's needed, without dictating anything about the development workflow, other than that you need to tell installers how to make an sdist if you want people to install stuff you shipped without an sdist or a wheel, or if you want to use any generic build-running tools that need to know your build hook(s). > development checkouts should be discouraged unless you're actively working on > that project. Perhaps Jim can chime in on this point, but when you work with a whole bunch of people developing a whole bunch of libraries making up a larger project (e.g. Zope), it doesn't seem very sensible to expect that everybody manually check out and manage all the dependencies they're using. Maybe you could mitigate that somewhat with some sort automated continuous build/release system, but that's not always a practical option. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Jim Fulton writes: > Anywhere you extract them is likely going to lead to access control > or security issues and generally cause pain, IMO. right! search the web for PYTHON_EGG_CACHE. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On Wed, Mar 27, 2013 at 4:50 PM, Vinay Sajip wrote: > Jim Fulton zope.com> writes: > >> It's a trap! >> >> At least on Unix systems: >> >> - Extensions in zip files that get magically extracted to a user's >> home directory lead to tragic deployment failures for services that >> run as special users. > > I can see how it would lead to problems, but the home directory location is > just as a proof of concept - the cache doesn't need to be in any private > place. Anywhere you extract them is likely going to lead to access control or security issues and generally cause pain, IMO. > >> - Zip files are a pain in the ass during development or debugging. > > Of course, but wheels are for deployment, not development, and this is one > possibility for deployment (several people have mentioned wanting to sometimes > just add wheels to sys.path rather than installing them, which got me thinking > about this functionality). I expect to use wheels during development just like I use eggs now. Not for development of the wheel/egg, but for development of something that uses it. You're in pdb and you land in a zipped egg/wheel that the package under development invoked and now you're screwed. > >> - Zip files are slower to import from (at least in my experience) > > It's just another option for a user of wheels. Caveat emptor, and all that. It's been tried with eggs. This is not new ground. Encouraging people to do this is going to cause pain and resentment. I think one of the reasons there's so much (IMO mostly irrational) hate for eggs is that people think you can only used zipped eggs, and zipped eggs cause pain and agony. > >> It would be far better IMO to just unzip the wheel and put that in >> your path. (I'm hoping that wheels used this way are a suitable >> replacement for eggs.) > > Well that's tantamount to installing the wheel, Not really. If you just unzip the wheel and add it to your path, you can stop using it by just removing from your path. If you install the wheel, it's contents will be poured into site-packages (and other places).It's much heavier than just adding the wheel (zipped or unzipped) to your path. > which is fine. I was thinking > along the line of egg replacement Me too. > - AFAIK eggs allow you to import extensions > from zip in a similar fashion. Importing from zipped eggs has proved itself to be an anti pattern. Buildout (as of buildout 2) always unzips eggs. It can then generate scripts with just the packages they need by adding (unzipped) eggs to sys.path. Various plugin systems (including buildout itself with extensions and recipes) do this dynamically at run time. It's very useful. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
Jim Fulton zope.com> writes: > It's a trap! > > At least on Unix systems: > > - Extensions in zip files that get magically extracted to a user's > home directory lead to tragic deployment failures for services that > run as special users. I can see how it would lead to problems, but the home directory location is just as a proof of concept - the cache doesn't need to be in any private place. > - Zip files are a pain in the ass during development or debugging. Of course, but wheels are for deployment, not development, and this is one possibility for deployment (several people have mentioned wanting to sometimes just add wheels to sys.path rather than installing them, which got me thinking about this functionality). > - Zip files are slower to import from (at least in my experience) It's just another option for a user of wheels. Caveat emptor, and all that. > It would be far better IMO to just unzip the wheel and put that in > your path. (I'm hoping that wheels used this way are a suitable > replacement for eggs.) Well that's tantamount to installing the wheel, which is fine. I was thinking along the line of egg replacement - AFAIK eggs allow you to import extensions from zip in a similar fashion. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Importable wheels using distlib/distil
On Wed, Mar 27, 2013 at 3:21 PM, Vinay Sajip wrote:
>> I'm not top-posting, but trying to keep GMane happy ...
>
> Since wheels are .zip files, they can sometimes be used to provide
> functionality without needing to be installed. Whereas .zip files contain no
> convention for indicating compatibility with a particular Python, wheels do
> contain this compatibility information. Thus, it is possible to check if a
> wheel can be directly imported from, and the wheel support in distlib allows
> you to take advantage of this using the mount() and unmount() methods. When
> you
> mount a wheel, its absolute path name is added to sys.path, allowing the
> Python
> code in it to be imported. (A DistlibException is raised if the wheel isn't
> compatible with the Python which calls the mount() method.)
>
> You don't need mount() just to add the wheel's name to sys.path, or to import
> pure-Python wheels. The mount() method goes further than just enabling Python
> imports - any C extensions in the wheel are also made available for import.
> For
> this to be possible, the wheel has to be built with additional metadata about
> extensions - a JSON file called EXTENSIONS which serialises an extension
> mapping dictionary. This maps extension module names to the names in the wheel
> of the shared libraries which implement those modules.
>
> Running unmount() on the wheel removes its absolute pathname from sys.path and
> makes its C extensions, if any, also unavailable for import.
>
> Wheels built with distil contain the EXTENSIONS metadata, so can be mounted
> complete with C extensions:
>
> $ distil download -d /tmp simplejson
> Downloading simplejson-3.1.2.tar.gz to /tmp/simplejson-3.1.2
> 63KB @ 73 KB/s 100 % Done: 00:00:00
> Unpacking ... done.
> $ distil package --fo=wh -d /tmp /tmp/simplejson-3.1.2/
> The following packages were built:
> /tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl
> $ python
> Python 2.7.2+ (default, Jul 20 2012, 22:15:08)
> [GCC 4.6.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
from distlib.wheel import Wheel
w = Wheel('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
w.mount()
import simplejson._speedups
dir(simplejson._speedups)
> ['__doc__', '__file__', '__loader__', '__name__', '__package__',
> 'encode_basestring_ascii', 'make_encoder', 'make_scanner', 'scanstring']
simplejson._speedups.__file__
> '/home/vinay/.distlib/dylib-cache/simplejson/_speedups.so'
>
> This, IMO, makes the wheel format more useful than it already is :-)
It's a trap!
At least on Unix systems:
- Extensions in zip files that get magically extracted to a user's
home directory lead to tragic deployment failures for services that
run as special users.
- Zip files are a pain in the ass during development or debugging.
- Zip files are slower to import from (at least in my experience)
It would be far better IMO to just unzip the wheel and put that in
your path. (I'm hoping that wheels used this way are a suitable
replacement for eggs.)
Jim
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 8:08 PM, Daniel Holth wrote: > We have a different definition of build tools if installing an sdist > that has a C extension doesn't make pip a build tool already. Then the word "build tool" is irrelevant, and the whole discussion of builders vs installers is pointless, since installers in the meaning most people use it within Python by necessity also are builders. The point is still that pip IMO should be a tool to *install* distributions, not *make* distributions. That's what is relevant. If the word "builder" does not describe the too that builds distribution then lets not use that word. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Importable wheels using distlib/distil
> I'm not top-posting, but trying to keep GMane happy ...
Since wheels are .zip files, they can sometimes be used to provide
functionality without needing to be installed. Whereas .zip files contain no
convention for indicating compatibility with a particular Python, wheels do
contain this compatibility information. Thus, it is possible to check if a
wheel can be directly imported from, and the wheel support in distlib allows
you to take advantage of this using the mount() and unmount() methods. When you
mount a wheel, its absolute path name is added to sys.path, allowing the Python
code in it to be imported. (A DistlibException is raised if the wheel isn't
compatible with the Python which calls the mount() method.)
You don't need mount() just to add the wheel's name to sys.path, or to import
pure-Python wheels. The mount() method goes further than just enabling Python
imports - any C extensions in the wheel are also made available for import. For
this to be possible, the wheel has to be built with additional metadata about
extensions - a JSON file called EXTENSIONS which serialises an extension
mapping dictionary. This maps extension module names to the names in the wheel
of the shared libraries which implement those modules.
Running unmount() on the wheel removes its absolute pathname from sys.path and
makes its C extensions, if any, also unavailable for import.
Wheels built with distil contain the EXTENSIONS metadata, so can be mounted
complete with C extensions:
$ distil download -d /tmp simplejson
Downloading simplejson-3.1.2.tar.gz to /tmp/simplejson-3.1.2
63KB @ 73 KB/s 100 % Done: 00:00:00
Unpacking ... done.
$ distil package --fo=wh -d /tmp /tmp/simplejson-3.1.2/
The following packages were built:
/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl
$ python
Python 2.7.2+ (default, Jul 20 2012, 22:15:08)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from distlib.wheel import Wheel
>>> w = Wheel('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
>>> w.mount()
>>> import simplejson._speedups
>>> dir(simplejson._speedups)
['__doc__', '__file__', '__loader__', '__name__', '__package__',
'encode_basestring_ascii', 'make_encoder', 'make_scanner', 'scanstring']
>>> simplejson._speedups.__file__
'/home/vinay/.distlib/dylib-cache/simplejson/_speedups.so'
>>>
This, IMO, makes the wheel format more useful than it already is :-)
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 2:50 PM, Lennart Regebro wrote: > On Wed, Mar 27, 2013 at 6:09 PM, Daniel Holth wrote: >> In my view the fact that pip creates an installation as an artifact of >> installing from a source package is equivalent to creating a wheel, >> given that wheel is a format defined as a zip file containing one >> installation of a distribution. Both operations equally ruin pip's >> reputation as being an installer instead of a build tool. > > How installing something can ruin the reputation as an installer is beyond me. > >> Instead all >> installation should have an intermediate, static, documented binary >> representation created by the build tool that is later moved into >> place by the install tool. I would be pleased if "pip install" lost >> the ability to natively install sdists without that intermediate step. > > That's a separate issue, but I disagree with that as well. > > //Lennart We have a different definition of build tools if installing an sdist that has a C extension doesn't make pip a build tool already. Clearly we're just going to disagree on this one. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 6:44 PM, Vinay Sajip wrote: > Lennart Regebro gmail.com> writes: > > >> Fine, as a stop-gap measure pip wheel might be useful, as this >> mythical packaging tool doesn't really exist yet (except as >> bdist_wheel, but I suspect pip wheel does more than that?) > > Well the distil tool does exist, and though I'm not claiming that it's ready > for > prime-time yet, it seems well on the way to being useful for this and other > purposes. Exactly. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 6:09 PM, Daniel Holth wrote: > In my view the fact that pip creates an installation as an artifact of > installing from a source package is equivalent to creating a wheel, > given that wheel is a format defined as a zip file containing one > installation of a distribution. Both operations equally ruin pip's > reputation as being an installer instead of a build tool. How installing something can ruin the reputation as an installer is beyond me. > Instead all > installation should have an intermediate, static, documented binary > representation created by the build tool that is later moved into > place by the install tool. I would be pleased if "pip install" lost > the ability to natively install sdists without that intermediate step. That's a separate issue, but I disagree with that as well. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 27, 2013, at 2:04 PM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > > >> I don't think the packaging formats should dictate the development flow at >> all. > > We might be at cross purposes here. If we posit that packaging metadata is in > JSON format (which I think we both favour), I was addressing Daniel's > objection > to it on the grounds that he doesn't like editing JSON, to suggest an > alternative > for people with that objection. It doesn't follow that they *have* to use any > particular workflow or tool, or that packaging formats are dictating it (other > than the bare fact that they are JSON). > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Gotcha, yea in my mind the JSON is generated by the archiver tool and added to the various types of dists, wheels, etc. What the users actually edit/use is totally up to the archiver tool. It could be .in files it could be a python file, it could be YAML, it could pull from a SQLite database. Packaging shouldn't care as long as it gets it's sdists bdists, wheels etc in the proper format with the proper metadata files. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
Donald Stufft stufft.io> writes: > I don't think the packaging formats should dictate the development flow at > all. We might be at cross purposes here. If we posit that packaging metadata is in JSON format (which I think we both favour), I was addressing Daniel's objection to it on the grounds that he doesn't like editing JSON, to suggest an alternative for people with that objection. It doesn't follow that they *have* to use any particular workflow or tool, or that packaging formats are dictating it (other than the bare fact that they are JSON). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
Agreed that python is a fine language for build scripts. On Wed, Mar 27, 2013 at 1:51 PM, Donald Stufft wrote: > > On Mar 27, 2013, at 1:41 PM, Vinay Sajip wrote: > >> PJ Eby telecommunity.com> writes: >> >>> The challenge here is again the distinction between raw source and >>> sdist, and the interaction with revision control. Either there has to >>> be some way to tell MEBS (i.e. the overall build system) what tool >>> you're using to generate that JSON, or you have to check a generated >>> file into revision control, and make sure you've updated it. (Which >>> is error prone, even if you don't mind checking generated files into >>> revision control.) >>> >>> This strongly suggests there needs to be *some* human-editable way to >>> at *least* specify what tool you're using to generate the JSON with. >> >> There are no doubt many possible workflows, but one such is: >> >> metadata input files - any number, hand-edited, checked into source control >> metadata merge tool - creates JSON metadata from input files >> JSON metadata - produced by tool, so not checked in >> >> If the "merge tool" (which could be a simple Python script) is custom to a >> project, it can be checked into source control in that project. If it is used >> across multiple projects, it is maintained as a separate tool in its own repo >> and, if you are just using it but not maintaining it, it becomes part of your >> build toolset (like sphinx-build). Actually, the doc tools seem to be a good >> analogy - create a useful format which is a pain to edit by hand (HTML that >> looks nice in a browser) from some checked in sources which are reasonable >> to edit by hand (.rst) + a merge tool (Sphinx). >> >> The merge tool seems similar in kind to the release.py script that many >> projects have, which creates release distribution files, bumps version >> numbers, >> registers and uploads to PyPI. >> >> Regards, >> >> Vinay Sajip >> >> ___ >> Distutils-SIG maillist - Distutils-SIG@python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > > I don't think the packaging formats should dictate the development flow at > all. .IN files and such all dictate how that should be. To me this is an > installer issue not a packaging issue and it's best solved in the installers. > Obviously there is some benefit to a "standard" way for installers to treat > these but I don't think it should be defined in terms of the packaging > formats. Hence my off the cuff suggestion of keeping setup.py develop, or > develop.py or some such script that express purpose is in use for development > checkouts, but that development checkouts should be discouraged unless you're > actively working on that project. > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 27, 2013, at 1:41 PM, Vinay Sajip wrote: > PJ Eby telecommunity.com> writes: > >> The challenge here is again the distinction between raw source and >> sdist, and the interaction with revision control. Either there has to >> be some way to tell MEBS (i.e. the overall build system) what tool >> you're using to generate that JSON, or you have to check a generated >> file into revision control, and make sure you've updated it. (Which >> is error prone, even if you don't mind checking generated files into >> revision control.) >> >> This strongly suggests there needs to be *some* human-editable way to >> at *least* specify what tool you're using to generate the JSON with. > > There are no doubt many possible workflows, but one such is: > > metadata input files - any number, hand-edited, checked into source control > metadata merge tool - creates JSON metadata from input files > JSON metadata - produced by tool, so not checked in > > If the "merge tool" (which could be a simple Python script) is custom to a > project, it can be checked into source control in that project. If it is used > across multiple projects, it is maintained as a separate tool in its own repo > and, if you are just using it but not maintaining it, it becomes part of your > build toolset (like sphinx-build). Actually, the doc tools seem to be a good > analogy - create a useful format which is a pain to edit by hand (HTML that > looks nice in a browser) from some checked in sources which are reasonable > to edit by hand (.rst) + a merge tool (Sphinx). > > The merge tool seems similar in kind to the release.py script that many > projects have, which creates release distribution files, bumps version > numbers, > registers and uploads to PyPI. > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig I don't think the packaging formats should dictate the development flow at all. .IN files and such all dictate how that should be. To me this is an installer issue not a packaging issue and it's best solved in the installers. Obviously there is some benefit to a "standard" way for installers to treat these but I don't think it should be defined in terms of the packaging formats. Hence my off the cuff suggestion of keeping setup.py develop, or develop.py or some such script that express purpose is in use for development checkouts, but that development checkouts should be discouraged unless you're actively working on that project. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 1:09 PM, Daniel Holth wrote: > I would be pleased if "pip install" lost > the ability to natively install sdists without that intermediate step. At that point, it would be giving easy_install (or any other tool that did) a comparative advantage. So that's probably not going to fly. (Unless of course you meant that the intermediate step remains transparent to the user.) easy_install (and pip) became popular because they get code from developers to users with the fewest possible steps for people on either end of the distribution channel. Adding pointless steps is both bad UI design and poor marketing. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
Lennart Regebro gmail.com> writes: > Fine, as a stop-gap measure pip wheel might be useful, as this > mythical packaging tool doesn't really exist yet (except as > bdist_wheel, but I suspect pip wheel does more than that?) Well the distil tool does exist, and though I'm not claiming that it's ready for prime-time yet, it seems well on the way to being useful for this and other purposes. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
PJ Eby telecommunity.com> writes: > The challenge here is again the distinction between raw source and > sdist, and the interaction with revision control. Either there has to > be some way to tell MEBS (i.e. the overall build system) what tool > you're using to generate that JSON, or you have to check a generated > file into revision control, and make sure you've updated it. (Which > is error prone, even if you don't mind checking generated files into > revision control.) > > This strongly suggests there needs to be *some* human-editable way to > at *least* specify what tool you're using to generate the JSON with. There are no doubt many possible workflows, but one such is: metadata input files - any number, hand-edited, checked into source control metadata merge tool - creates JSON metadata from input files JSON metadata - produced by tool, so not checked in If the "merge tool" (which could be a simple Python script) is custom to a project, it can be checked into source control in that project. If it is used across multiple projects, it is maintained as a separate tool in its own repo and, if you are just using it but not maintaining it, it becomes part of your build toolset (like sphinx-build). Actually, the doc tools seem to be a good analogy - create a useful format which is a pain to edit by hand (HTML that looks nice in a browser) from some checked in sources which are reasonable to edit by hand (.rst) + a merge tool (Sphinx). The merge tool seems similar in kind to the release.py script that many projects have, which creates release distribution files, bumps version numbers, registers and uploads to PyPI. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 1:30 PM, Donald Stufft wrote: > > On Mar 27, 2013, at 1:12 PM, PJ Eby wrote: > >> On Wed, Mar 27, 2013 at 8:02 AM, Vinay Sajip wrote: >>> To those people who would balk at editing JSON by hand - who's asking you >>> to? Why not just get the data into an appropriate dict, using any tools you >>> like, and then serialise it to JSON? >> >> The challenge here is again the distinction between raw source and >> sdist, and the interaction with revision control. Either there has to >> be some way to tell MEBS (i.e. the overall build system) what tool >> you're using to generate that JSON, or you have to check a generated >> file into revision control, and make sure you've updated it. (Which >> is error prone, even if you don't mind checking generated files into >> revision control.) >> >> This strongly suggests there needs to be *some* human-editable way to >> at *least* specify what tool you're using to generate the JSON with. >> ___ >> Distutils-SIG maillist - Distutils-SIG@python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > I don't actually think packaging needs to solve this. But there are a number > of solutions that come to mind (mostly either expecting a standard command > ala setup.py develop to work). > > If I want to install a development version of say libsodium (just an example > C lib) I download it and run ./autogen.sh && make make install but once it's > packaged I can install it using the packaging tools. > > So this issue is really sort of parallel to builders, archivers and even the > JSON and it comes down to how does an unpackaged directory of code (the VCS > checkout portion isn't really that important here) signal to an installer how > to install a development version of it. Personally I think a common > entrypoint (ala make install) is the way forward for this. When you leave the > realm of package formats (ala sdist, wheel, etc) you start needing to get > much more freeform. It does get a little murky. nothing the file in a source checkout PKG-INFO the file in an sdist PKG-INFO the re-generated file PKG-INFO the installed file (we will probably call it metadata.json soon but the confusion is the same). I think it might make sense to expect only a stub PKG-INFO[.in] at the root of a VCS checkout, have a 100% generated and hopefully trustworthy .dist-info directory in an sdist, and don't bother regenerating the root PKG-INFO. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 27, 2013, at 1:12 PM, PJ Eby wrote: > On Wed, Mar 27, 2013 at 8:02 AM, Vinay Sajip wrote: >> To those people who would balk at editing JSON by hand - who's asking you >> to? Why not just get the data into an appropriate dict, using any tools you >> like, and then serialise it to JSON? > > The challenge here is again the distinction between raw source and > sdist, and the interaction with revision control. Either there has to > be some way to tell MEBS (i.e. the overall build system) what tool > you're using to generate that JSON, or you have to check a generated > file into revision control, and make sure you've updated it. (Which > is error prone, even if you don't mind checking generated files into > revision control.) > > This strongly suggests there needs to be *some* human-editable way to > at *least* specify what tool you're using to generate the JSON with. > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig I don't actually think packaging needs to solve this. But there are a number of solutions that come to mind (mostly either expecting a standard command ala setup.py develop to work). If I want to install a development version of say libsodium (just an example C lib) I download it and run ./autogen.sh && make make install but once it's packaged I can install it using the packaging tools. So this issue is really sort of parallel to builders, archivers and even the JSON and it comes down to how does an unpackaged directory of code (the VCS checkout portion isn't really that important here) signal to an installer how to install a development version of it. Personally I think a common entrypoint (ala make install) is the way forward for this. When you leave the realm of package formats (ala sdist, wheel, etc) you start needing to get much more freeform. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/26/2013 09:12 PM, PJ Eby wrote: > (Some tools do check for the *existence* of a PKG-INFO, like PyPI's > sdist upload validation, and the various egg formats require a file > *named* PKG-INFO, but AFAIK nothing commonly used out there actually > *reads* PKG-INFO or gives a darn about its contents, except for that > version usecase mentioned above.) I do have a tool (named 'pkginfo', funnily enough) which does parse them: https://pypi.python.org/pypi/pkginfo http://pythonhosted.org/pkginfo/ I use it in another tool, 'compoze', which allows me to build "cureated" indexes from versions installed locally (e.g., after testing in a virtualenv): https://pypi.python.org/pypi/compoze/ http://docs.repoze.org/compoze/ Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFTKBcACgkQ+gerLs4ltQ4x/wCfZsp/p60ELrQvTCXfdPMhuK1E qJQAoJXvTlTSo1iy/KxylnuizPodbr25 =IJ6t -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 8:02 AM, Vinay Sajip wrote: > To those people who would balk at editing JSON by hand - who's asking you > to? Why not just get the data into an appropriate dict, using any tools you > like, and then serialise it to JSON? The challenge here is again the distinction between raw source and sdist, and the interaction with revision control. Either there has to be some way to tell MEBS (i.e. the overall build system) what tool you're using to generate that JSON, or you have to check a generated file into revision control, and make sure you've updated it. (Which is error prone, even if you don't mind checking generated files into revision control.) This strongly suggests there needs to be *some* human-editable way to at *least* specify what tool you're using to generate the JSON with. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 12:18 PM, Lennart Regebro wrote: > On Wed, Mar 27, 2013 at 4:49 PM, Daniel Holth wrote: >> Then you are also in favor of removing sdist support from the "pip >> install" command, in the same way that rpm doesn't automatically >> compile srpm. > > I was not aware that pip could create sdists. In my view the fact that pip creates an installation as an artifact of installing from a source package is equivalent to creating a wheel, given that wheel is a format defined as a zip file containing one installation of a distribution. Both operations equally ruin pip's reputation as being an installer instead of a build tool. Instead all installation should have an intermediate, static, documented binary representation created by the build tool that is later moved into place by the install tool. I would be pleased if "pip install" lost the ability to natively install sdists without that intermediate step. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 4:49 PM, Daniel Holth wrote: > Then you are also in favor of removing sdist support from the "pip > install" command, in the same way that rpm doesn't automatically > compile srpm. I was not aware that pip could create sdists. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 27, 2013, at 10:57 AM, Vinay Sajip wrote: > Lennart Regebro gmail.com> writes: > >> It makes no sense to have a tools for developers that does everything >> including running building, running tests and packaging, and another >> tool that does nothing but installs, and creates wheel packages. >> >> Making wheels should be a part of the tool using for packaging, not >> the tool used for installing. > > Don't forget that developers are users too - they consume packages as well as > developing them. I see no *conceptual* harm in a tool that can do > archive/build/install, as long as it can do them well (harder to do than to > say, > I know). And > I see that there is a place for just-installation functionality which does not > require the presence of a build environment. But a single tool could have > multiple guises, just as some Unix tools of old behaved differently according > to > which link they were invoked from (the linked-to executable being the same). > > Isn't our present antagonism to the idea of having one ring to bind them all > due to the qualities specific to that ring (setup.py, calls to setup())? > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Basically this. There no need to *enforce* that the toolchain each be separate pieces, but rather ensure that it *can*. The current status quo means setuptools (or distutils) are the only name in the game, if you want to do anything else you have to pretend you are setuptools. In short setuptools owns the entire process. The goal here is to break it up so no one tool owns the entire process, but still allow tools to act as more then one part of the process when it makes sense. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 11:22 AM, Lennart Regebro wrote: > On Wed, Mar 27, 2013 at 3:47 PM, Vinay Sajip wrote: >> One likely scenario on Windows is that you have a compiler and can install >> from >> sdists or wheels, but want to distribute packages to people who don't have a >> compiler, so can only install from wheels. > > Which means you are actually not just a simple end user, but ops or > devops who want to build packages. And then the question arises why we > can't have documentation explaining how to build packages with the > packaging tools that is usable for that user. > > Fine, as a stop-gap measure pip wheel might be useful, as this > mythical packaging tool doesn't really exist yet (except as > bdist_wheel, but I suspect pip wheel does more than that?) > But in the long run I don't see the point, and I think it muddles what > pip is and does. Then you are also in favor of removing sdist support from the "pip install" command, in the same way that rpm doesn't automatically compile srpm. Pip wheel does nothing more than run bdist_wheel on each package in a requirements set. It's kindof a stopgap measure but it's also a firm foundation for the more decoupled way packaging should work. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 3:47 PM, Vinay Sajip wrote: > One likely scenario on Windows is that you have a compiler and can install > from > sdists or wheels, but want to distribute packages to people who don't have a > compiler, so can only install from wheels. Which means you are actually not just a simple end user, but ops or devops who want to build packages. And then the question arises why we can't have documentation explaining how to build packages with the packaging tools that is usable for that user. Fine, as a stop-gap measure pip wheel might be useful, as this mythical packaging tool doesn't really exist yet (except as bdist_wheel, but I suspect pip wheel does more than that?) But in the long run I don't see the point, and I think it muddles what pip is and does. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On 27 March 2013 14:41, Lennart Regebro wrote: > Can you explain to me why you as an end user can not just install the > packages? > Why do you need to first build wheels? Mainly just as Daniel said for convenience of repeat installs (in virtualenvs). But also I think there are a *lot* of different workflows out there and we need to avoid focusing on any one exclusively (the strict builder/installer split is more focused on production installs than on developers installing into virtualenvs, for instance). On 27 March 2013 14:44, Vinay Sajip wrote: > I personally have no problem with "pip wheel" staying, but it does muddy pip's > original intent as denoted by pip standing for "pip installs packages". I think we have to remember that pip is a reasonably mature tool with a large existing user base. I don't want the idea that pip is now the "official Python installer" to be at odds with continued support of those users and their backward compatibility needs. Refactoring pip's internals and moving towards support of the new standards and workflow models is one thing, and I'm 100% in favour of that, but I don't see major changes in fundamentals like "pip install foo" (working seamlessly even if there are no wheels available for foo) being on the cards. Having a "pip wheel" command fits into that model simply as a way of saying "stop the pip install process just before the actual final install, so that I can run that final step over and over without redoing the first part". Think of it as "pip install --all-but-install" if you like :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 10:57 AM, Vinay Sajip wrote: > Lennart Regebro gmail.com> writes: > >> It makes no sense to have a tools for developers that does everything >> including running building, running tests and packaging, and another >> tool that does nothing but installs, and creates wheel packages. >> >> Making wheels should be a part of the tool using for packaging, not >> the tool used for installing. > > Don't forget that developers are users too - they consume packages as well as > developing them. I see no *conceptual* harm in a tool that can do > archive/build/install, as long as it can do them well (harder to do than to > say, > I know). And > I see that there is a place for just-installation functionality which does not > require the presence of a build environment. But a single tool could have > multiple guises, just as some Unix tools of old behaved differently according > to > which link they were invoked from (the linked-to executable being the same). > > Isn't our present antagonism to the idea of having one ring to bind them all > due to the qualities specific to that ring (setup.py, calls to setup())? I really think so. distutils is a bad implementation. This has a lot more to do with how it works internally than how its command line interface looks. We can have new tools that do everything with a single command but really delegate the work out to separate decoupled and hopefully pluggable pieces underneath. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
Lennart Regebro gmail.com> writes: > It makes no sense to have a tools for developers that does everything > including running building, running tests and packaging, and another > tool that does nothing but installs, and creates wheel packages. > > Making wheels should be a part of the tool using for packaging, not > the tool used for installing. Don't forget that developers are users too - they consume packages as well as developing them. I see no *conceptual* harm in a tool that can do archive/build/install, as long as it can do them well (harder to do than to say, I know). And I see that there is a place for just-installation functionality which does not require the presence of a build environment. But a single tool could have multiple guises, just as some Unix tools of old behaved differently according to which link they were invoked from (the linked-to executable being the same). Isn't our present antagonism to the idea of having one ring to bind them all due to the qualities specific to that ring (setup.py, calls to setup())? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 10:41 AM, Lennart Regebro wrote: > On Wed, Mar 27, 2013 at 3:16 PM, Paul Moore wrote: >> But sometimes practicality beats purity. As an end user who wants to >> just install packages, but who knows that not everything will be >> available as wheels, I need to be able to build my own wheels. > > Can you explain to me why you as an end user can not just install the > packages? > Why do you need to first build wheels? > > //Lennart It's because when you install lots of the same packages repeatedly you might want it to be lightning fast the second time. The pip wheel workflow also gives you a useful local copy of all the packages you need, insulating yourself from pypi outages. This is the practical side. The long term / bigger picture use case is that the wheel format or an equivalent manifest serves as a sort of packaging WSGI analogue -- a static interface between builds and installs. We would remove the "setup.py install" command entirely. In that world pip would have to build the wheel because it couldn't "just install" the package. The first convenient wheel tool was more like wheeler.py. It was just a shell script that called pip install --no-install, and "setup.py bdist_wheel for subdirectory in build directory". ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
Lennart Regebro gmail.com> writes: > > Can you explain to me why you as an end user can not just install the > packages? > Why do you need to first build wheels? > One likely scenario on Windows is that you have a compiler and can install from sdists or wheels, but want to distribute packages to people who don't have a compiler, so can only install from wheels. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
Paul Moore gmail.com> writes: > Of course if someone builds a "wheelmaker" tool that did precisely > what "pip wheel" did, I would have no objections to using that. But I already have made one, it's called wheeler.py [1]. It uses vanilla pip (not the variant which provides pip wheel) to build wheels from sdists on PyPI. The distil tool builds wheels with or without using vanilla pip as a helper; the vanilla pip helper is needed where you *have* to run setup.py to get a correct build (not always the case). With wheeler.py you need to install distlib, while with distil it's included. > even then, the mere existence of another tool doesn't seem to me to be > enough justification for removing functionality from pip. If pip wheel > didn't exist, and someone had written wheelmaker, I would not be > arguing to *add* pip wheel. But it's there already and there's a much > higher bar for removing useful functionality. I personally have no problem with "pip wheel" staying, but it does muddy pip's original intent as denoted by pip standing for "pip installs packages". While "pip wheel" was added as a pragmatic way of getting wheels out there for people to work with, pip's wheel functionality has only recently been added and is unlikely to be widespread, e.g. in distro packages for pip. So it could be reverted (since there are alternatives) and ISTM that the likely impact would only be on a few early adopters. Note that I'm not arguing for reversion at all - it makes sense for there for multiple implementations of wheel building and usage so that interoperability wrinkles can be ironed out. Regards, Vinay Sajip [1] https://gist.github.com/vsajip/4988471 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 3:16 PM, Paul Moore wrote: > But sometimes practicality beats purity. As an end user who wants to > just install packages, but who knows that not everything will be > available as wheels, I need to be able to build my own wheels. Can you explain to me why you as an end user can not just install the packages? Why do you need to first build wheels? //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 10:04 AM, Lennart Regebro wrote: > On Wed, Mar 27, 2013 at 2:57 PM, Daniel Holth wrote: >> Is it too convenient? The tool knows how to find sources, compile >> them, and install them. It will delegate all the work to the actual >> build system. If pip was a pure installer without a way to invoke a >> build system then it wouldn't be able to install from sdist at all. > > All of that should be implemented in a library that pip can use. So > this is only a question of a conceptual difference between different > tools. > > It makes no sense to have a tools for developers that does everything > including running building, running tests and packaging, and another > tool that does nothing but installs, and creates wheel packages. > > Making wheels should be a part of the tool using for packaging, not > the tool used for installing. It kindof works this way already. Pip doesn't include any of the actual wheel building logic, it just collects the necessary sources then calls out to the "build one wheel" tool for each downloaded source archive. The developer's tool has some overlap in functionality but is focused on dealing with one first-party package at a time for upload to the index rather than many packages at a time for download and install. What will change is that pip will include the install logic instead of delegating it to the worst shortcoming of packaging "setup.py install". ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On 27 March 2013 14:04, Lennart Regebro wrote: > On Wed, Mar 27, 2013 at 2:57 PM, Daniel Holth wrote: >> Is it too convenient? The tool knows how to find sources, compile >> them, and install them. It will delegate all the work to the actual >> build system. If pip was a pure installer without a way to invoke a >> build system then it wouldn't be able to install from sdist at all. > > All of that should be implemented in a library that pip can use. So > this is only a question of a conceptual difference between different > tools. > > It makes no sense to have a tools for developers that does everything > including running building, running tests and packaging, and another > tool that does nothing but installs, and creates wheel packages. > > Making wheels should be a part of the tool using for packaging, not > the tool used for installing. But sometimes practicality beats purity. As an end user who wants to just install packages, but who knows that not everything will be available as wheels, I need to be able to build my own wheels. But I don't want a full development tool. Having the install tool able to do a download and build from sdist is a huge convenience to me. Of course if someone builds a "wheelmaker" tool that did precisely what "pip wheel" did, I would have no objections to using that. But even then, the mere existence of another tool doesn't seem to me to be enough justification for removing functionality from pip. If pip wheel didn't exist, and someone had written wheelmaker, I would not be arguing to *add* pip wheel. But it's there already and there's a much higher bar for removing useful functionality. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 2:57 PM, Daniel Holth wrote: > Is it too convenient? The tool knows how to find sources, compile > them, and install them. It will delegate all the work to the actual > build system. If pip was a pure installer without a way to invoke a > build system then it wouldn't be able to install from sdist at all. All of that should be implemented in a library that pip can use. So this is only a question of a conceptual difference between different tools. It makes no sense to have a tools for developers that does everything including running building, running tests and packaging, and another tool that does nothing but installs, and creates wheel packages. Making wheels should be a part of the tool using for packaging, not the tool used for installing. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 9:39 AM, Lennart Regebro wrote: > On Tue, Mar 26, 2013 at 12:07 AM, Daniel Holth wrote: >> Unix users will always want to compile their own. > > Yup. > >> Pip wheel is not going away > > I don't see how that follows. > > //Lennart Is it too convenient? The tool knows how to find sources, compile them, and install them. It will delegate all the work to the actual build system. If pip was a pure installer without a way to invoke a build system then it wouldn't be able to install from sdist at all. It would help if you'd describe the alternative workflow again. The proposed workflow is "pip wheel package"; "pip install --find-links wheelhouse --no-index package". We don't suggest uploading the wheels used to cache compilation to pypi especially since most of them are probably other people's packages. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Tue, Mar 26, 2013 at 12:07 AM, Daniel Holth wrote: > Unix users will always want to compile their own. Yup. > Pip wheel is not going away I don't see how that follows. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mon, Mar 25, 2013 at 10:08 PM, Paul Moore wrote: > There's a longer-term issue that occurred to me when thinking about > pip's role as a "builder" or an "installer" (to use Nick's > terminology). > > As I understand Nick's vision for the future, installers (like pip) > will locate built wheels and download and install them, and builders > (like distutils and bento) will be responsible for building wheels. > But there's an intermediate role which shouldn't get forgotten in the > transition - the role that pip currently handles with the "pip wheel" > command. This is where I specify a list of distributions, and pip > locates sdists, downloads them, checks dependencies, and ultimately > builds all of the wheels. I'm not sure whether the current idea of > builders includes this "locate, download and resolve dependencies" > function (distutils and bento certainly don't have that capability). Personally I don't see that as an intermediate role at all. That for me is a builder. > I imagine that pip will retain some form of the current "pip wheel" I hope it will not. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] python setup.py bdist_rpm fails on RHEL5 x86_64
Hello everyone: I'm attempting to use the bdist_rpm flag to build a plain vanilla, Python 2.7 RPM for RHEL 5 x86_64, but the build command fails. Since you all are the distutils experts I figured you might have seen this before. I also submitted a bug to bugs.python.org: http://bugs.python.org/issue17553? Anyone have some pointers on how to make this build work? thanks Sean ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Fwd: python setup.py bdist_rpm fails on RHEL5 x86_64
My apologies if this comes through twice; I think I sent the first copy before I was approved to send to this list! ** Hello everyone: I'm attempting to use the bdist_rpm flag to build a plain vanilla, Python 2.7 RPM for RHEL 5 x86_64, but the build command fails. Since you all are the distutils experts I figured you might have seen this before. I also submitted a bug to bugs.python.org: http://bugs.python.org/issue17553? Anyone have some pointers on how to make this build work? thanks Sean ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 8:18 AM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > >> We need approximately five fields: >> >> Name >> Version >> Provides-Extra >> Requires-Dist >> Setup-Requires-Dist >> >> the rest are useless, never need to be parsed by anyone, or are >> already sent to pypi as a dict. >> >> We need the environment markers language. >> >> We need the requirements specifiers >= 4.0.0, < 9. >> > > You're taking a disappointingly narrow view of the metadata, it seems to me. > If > you look at the totality of metadata which describes distributions in the > here- > and-now world of setuptools, it's a lot more than that - just look at any of > my metadata files which I've pointed to in the "distil" documentation. > > You're only talking about *installation* metadata, but even there your > coverage > is incomplete. I won't go into any more details now, but suffice to say that > as > I am working on "distil", I am coming across decisions about installation > which > either I hard-code into distil (thus making it quite likely that another tool > will give different results), or enshrine in installation metadata > (constraining > all compliant tools to adhere to the developer's and/or user's wishes in that > area). Hooray for JSON. I actually liked the separation and viewed it as a de-coupling feature, but that will probably be less important as we avoid setup.py generating different metadata for each execution. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
Nick Coghlan gmail.com> writes: > It all adds up to it making far more sense to just switch the format to JSON for 2.0 rather than persisting with ad hoc attempts to use a key-value multidict for structured data storage. +1, and I sincerely hope you will take a look at the JSON metadata used in distlib/distil to good advantage in dependency resolution, and installing, archiving and building distributions. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
Daniel Holth gmail.com> writes: > We need approximately five fields: > > Name > Version > Provides-Extra > Requires-Dist > Setup-Requires-Dist > > the rest are useless, never need to be parsed by anyone, or are > already sent to pypi as a dict. > > We need the environment markers language. > > We need the requirements specifiers >= 4.0.0, < 9. > You're taking a disappointingly narrow view of the metadata, it seems to me. If you look at the totality of metadata which describes distributions in the here- and-now world of setuptools, it's a lot more than that - just look at any of my metadata files which I've pointed to in the "distil" documentation. You're only talking about *installation* metadata, but even there your coverage is incomplete. I won't go into any more details now, but suffice to say that as I am working on "distil", I am coming across decisions about installation which either I hard-code into distil (thus making it quite likely that another tool will give different results), or enshrine in installation metadata (constraining all compliant tools to adhere to the developer's and/or user's wishes in that area). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
Donald Stufft stufft.io> writes:
> I disagree.
>
> - These files are used for tools to exchange data, so "hand editable"
shouldn't be a primary concern.
Right. Nobody hand-edits PKG-INFO now, do they?
> - There are a number of current fields where the current format is *not*
enough and one off psuedo formats
> have had to be added
> - `Keywords: dog puppy voting election` - A list masquerading as a string,
this one needs field.split() to
> actually parse it
> - `Project-URL: Bug, Issue Tracker,
http://bitbucket.org/tarek/distribute/issues/` - A dictionary
> masquerading as a list of strings, this one needs {key.strip(): value.strip()
for key, value in
> [x.rsplit(", ", 1) for x in field]}
> - Any of the fields can contain arbitrary content, previously Description
had specialized handling for
> this which it has now been moved to the payload section, but all the same
issues there affect other fields.
> - The Extension field name using ExtensionName/ActualKey to kludge a nested
dictionary
> - The ExtensionName/json is a horrible kludge why are we nesting a format
inside of a format instead of just
> using a format that supports everything we could want?
>
> As far as I can tell the only things that even use PKG-INFO is
setuptools/distribute and we want to phase them
> out of existence anyways. The only other thing I can think of is Wheel which
can either a) be updated to a
> different format it's new enough there's not much need to worry about legacy
support or b) generate the
> METADATA file just for Wheels.
Please note that:
* I already have a system working fairly well *now* (though it's early days,
and needs more testing) where JSON is used for metadata.
* The metadata covers not just the index metadata (PKG-INFO) but also metadata
covering how to build, install and test distributions.
* The metadata already exists for the vast bulk of distributions on PyPI and
is derived from the setup.py in those distributions. So migration is not
much of an issue.
* The "distil" tool demonstrates each of the Archiver, Builder and Installer
roles reasonably well for its stage of development.
Donald's above analysis resonates with me - it seems pretty kludgy trying to
shoe-horn stuff into key-value format which doesn't fit it well. There don't
seem to be any valid technical arguments to keeping the key-value format, other
than "please let's not try to change too many things at once". If that's really
going to be accepted as the reason, it strikes me as being a little timid
(given what "distil" shows is possible). And that would be enough of a shame as
it is, without making things worse by introducing something like
ExtensionName/json.
To those people who would balk at editing JSON by hand - who's asking you
to? Why not just get the data into an appropriate dict, using any tools you
like, and then serialise it to JSON? That approach seems to be what JSON was
designed for. If any tools need PKG-INFO style metadata, that's easy enough to
generate from a JSON format, as distil's wheel building support demonstrates.
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
