Re: [Distutils] Is build an inherently arbitrary-code process?
On Thu, Mar 27, 2014 at 2:23 PM, Nick Coghlan ncogh...@gmail.com wrote: On 28 Mar 2014 05:42, Daniel Holth dho...@gmail.com wrote: I became convinced that build was an inherently arbitrary-code process, and not something to be universally handled by a declarative system, It wasn't an accident that last years PyCon panel was subtitled setup.py *install* must die :) As others have suggested, declarative build will never be more than an 80% solution, and then beyond that, it should be a question of plumbing to invoke the project's own build system (and ideally that plumbing will be per build system, not per project). Agreed -- I have been poking at this a bit, and trying to make gattai work for me: http://sourceforge.net/projects/gattai/ (It's a by and for python build system that essentially invokes the native build systems : make, setup.py, nmake, etc... But that's not really the point). the point is that I hit a wall pretty quickly with its declarative approach (JSON files). I find I can't do what I need to do with straight declaration (and I'm hacking gattai to support that) However, for the most part, all I need to be able to do is run arbitrary code to set up the declarations. The stuff that's been talked about: finding the right libraries to link to, that sort of thing. I actually think that aspect of the setup.py approach works pretty well, even though it does sometimes get kind of messy. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Is build an inherently arbitrary-code process?
On Mar 27, 2014, at 3:42 PM, Daniel Holth dho...@gmail.com wrote: I became convinced that build was an inherently arbitrary-code process, and not something to be universally handled by a declarative system, when I observed an autotools project under configuration. The things spend ten minutes writing and compiling snippets of C code to determine which features are and are not supported by the runtime. This is *very* arbitrary code and probably one of the tamer things people like to do during builds. As usual I blame distutils, because the separation is not there. So when you blame setup.py for having to run procedural code to generate the *metadata* you might go to far and think you should also eliminate a procedural build - simply because build and metadata are not adequately separate in the distutils design. Declarative build systems are a nice idea but they are not going to work for everyone. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig Well for the record auto tools does all of that because there’s near zero ability to introspect the compiler to figure out what it supports except by compiling little programs that depend on X feature and then running them to see if they work or not. That being said, yes building code is inherently a process that needs to execute things and cannot be made completely static. I believe the metadata itself can be static, and instead of an arbitrary build script we can declare a build hook, but that’s an argument for a PEP! I do believe a declarative build system can work for the 90% case though and should probably be the “default” option. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Is build an inherently arbitrary-code process?
On 27 March 2014 19:48, Donald Stufft don...@stufft.io wrote: I do believe a declarative build system can work for the 90% case though and should probably be the default option. +1. Easy things should be easy, and hard things possible. Ultimately, I want to say this is my project, here are the Python files. And oh, here's a couple of C files, nothing clever here, move along. When I interface with an external library that may not be in the same place on every system, I'd expect to say a bit more, but mainly just look in location X, but the user can override this with a command line argument. Going beyond that - autodetecting library locations, languages other than C (for example Cython), detecting CPU capabilities, etc - I would expect to be able to do it if I needed to, but I would not expect the declarative framework to be catering for those cases, I'd assume I needed to write code. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Is build an inherently arbitrary-code process?
On Mar 27, 2014, at 4:19 PM, Paul Moore p.f.mo...@gmail.com wrote: On 27 March 2014 19:48, Donald Stufft don...@stufft.io wrote: I do believe a declarative build system can work for the 90% case though and should probably be the default option. +1. Easy things should be easy, and hard things possible. Ultimately, I want to say this is my project, here are the Python files. And oh, here's a couple of C files, nothing clever here, move along. When I interface with an external library that may not be in the same place on every system, I'd expect to say a bit more, but mainly just look in location X, but the user can override this with a command line argument”. That’s the job of the linker, it has built in locations to look for libraries and env vars/command line flags to override that. Someone packaging a C-ext that needs a library to compile shouldn’t be specifying where to find it anyways. Going beyond that - autodetecting library locations, languages other than C (for example Cython), detecting CPU capabilities, etc - I would expect to be able to do it if I needed to, but I would not expect the declarative framework to be catering for those cases, I'd assume I needed to write code. Paul - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Is build an inherently arbitrary-code process?
On 27 March 2014 20:30, Donald Stufft don...@stufft.io wrote: When I interface with an external library that may not be in the same place on every system, I'd expect to say a bit more, but mainly just look in location X, but the user can override this with a command line argument. That's the job of the linker, it has built in locations to look for libraries and env vars/command line flags to override that. Someone packaging a C-ext that needs a library to compile shouldn't be specifying where to find it anyways. Ideally, yes. But not in practice (at least not on Windows). This is getting off-topic, so I'll just give a quick example and leave it at that. I recently wanted to build PyYAML on 64-bit Windows, which uses libyaml if that's available. First thing I had to do therefore was to build libyaml. That's not too hard (considering). But the build leaves include files in the project directory and a lib file somewhere in the bowels of the project ...\x64\Release\Output\Lib\libyaml.lib, IIRC). There is no install step, largely because there is nowhere standard to install include and lib files *to*. OK, so now I build PyYAML. I don't have the compiler, linker etc on my PATH - the distutils code automatically detects that I have the right version of the compiler installed and sets up the environment for me. This is a *feature*, not a problem, as otherwise I would have to remember which compiler to use for which Python version, and activate it as an extra step. As a result of this fact, though, I can't set LIB or INCLUDE environment variables to point to the libyaml project directories (nor would I particularly want to, as this is purely a one-off build). So I need to tell the build system (distutils) where to find libyaml and yaml.h. There's no way setup.py can guess where they might be in the general case, so I use the build_ext -L and -I options to specify. Consider the linker can just find stuff as the look in location X scenario. And the above as the reason we might need build-system support for adding compiler/linker flags or env vars at build time. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Is build an inherently arbitrary-code process?
On Thu, Mar 27, 2014 at 7:48 PM, Donald Stufft don...@stufft.io wrote: On Mar 27, 2014, at 3:42 PM, Daniel Holth dho...@gmail.com wrote: I became convinced that build was an inherently arbitrary-code process, and not something to be universally handled by a declarative system, when I observed an autotools project under configuration. The things spend ten minutes writing and compiling snippets of C code to determine which features are and are not supported by the runtime. This is *very* arbitrary code and probably one of the tamer things people like to do during builds. As usual I blame distutils, because the separation is not there. So when you blame setup.py for having to run procedural code to generate the *metadata* you might go to far and think you should also eliminate a procedural build - simply because build and metadata are not adequately separate in the distutils design. Declarative build systems are a nice idea but they are not going to work for everyone. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig Well for the record auto tools does all of that because there’s near zero ability to introspect the compiler to figure out what it supports except by compiling little programs that depend on X feature and then running them to see if they work or not. That being said, yes building code is inherently a process that needs to execute things and cannot be made completely static. I believe the metadata itself can be static, and instead of an arbitrary build script we can declare a build hook, but that’s an argument for a PEP! I do believe a declarative build system can work for the 90% case though and should probably be the “default” option. It is indeed possible for most python packages, and I tried to figure out quite a few things to make that percentage as high as possible in Bento. The really difficult part is not that, though, but making the system pluggable to allow pretty much arbitrary extensibility. What makes distutils very ill suited to that task is the whole subcommand model where options are not known in advance (that was the #1 issue when integrating distutils and scons). For bento, I came up with a system of 'build manifest', where each command produces a text-based 'protocol' that other commands might use ( https://github.com/pv/bento/blob/master/doc/source/hacking.rst). The implementation is not great but I think the idea is still sound: this means that in theory, you could build C extensions with something not involving python at all (e.g. make), but as long as the build process output that file, you could build installers from the system. IOW, the json file acts as 'middleware'. What any future solution should *not* try to do is inventing yet another build system David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Is build an inherently arbitrary-code process?
On 28 Mar 2014 05:42, Daniel Holth dho...@gmail.com wrote: I became convinced that build was an inherently arbitrary-code process, and not something to be universally handled by a declarative system, It wasn't an accident that last years PyCon panel was subtitled setup.py *install* must die :) As others have suggested, declarative build will never be more than an 80% solution, and then beyond that, it should be a question of plumbing to invoke the project's own build system (and ideally that plumbing will be per build system, not per project). Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
