Re: [Distutils] waf can be used to compile Python extension modules
On 24 Mar 2014 10:01, Paul Moore p.f.mo...@gmail.com wrote: On 23 March 2014 23:13, Nick Coghlan ncogh...@gmail.com wrote: Now you begin to see the scope of the problem. It's definitely solvable, but means asking a whole pile of required, recommended or distutils-specific? questions about the existing distutils and setuptools build system :) pip already relies on it sets the minimum for the required category, but there's more to a full build system abstraction than what pip currently supports. OK, I see now. So the ultimate build system will include pip changes to supply build-time options in an as-yet unspecified manner. There's certainly no way I can do all of that myself, I don't have remotely the level of experience with complex build requirements. But I can probably take the first steps, and leave it to people with the experience to add to it. No promises on timescales but I'll see what I can do. One thought. do we want to use a setup.py script as the interface, with all its historical baggage, or would we be better using a new script name as the official interface (with pip falling back to equivalent setup.py invocations when that script isn't present, for backward compatibility)? Step 1 is What does pip currently expect setup.py to support? Step 2 is What other existing features of distutils/setuptools do we think a reasonable replacement for setup.py should support? (I don't believe distutils2 reached the point of addressing this, but that should still be checked) Step 3 (what, if anything, should replace the setup.py CLI as the build system abstraction?) really depends on the outcome of steps 1 2 - this is more a research/documentation consolidation project at this point than it is a design project. Cheers, Nick. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
Step 3 (what, if anything, should replace the setup.py CLI as the build system abstraction?) [...] research/documentation consolidation project at this point than it is a design project. FYI, I've been experimenting with a design approach for a build system in distil. Currently it only covers what distutils/setuptools already do - i.e. build_py functionality + build_ext functionality covering libraries, extensions, setuptools Features, Cython / Pyrex and Fortran / f2py. Not too shabby, but clearly that's not enough. Instead of compiler classes (the abstractions used in distutils / setuptools), distil's approach focuses on a command line with variable placeholders - this allows just about anything to be plugged in for custom builds. There's still work to be done on it, but the initial approach looks promising. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On 24 Mar 2014 00:13, Paul Moore p.f.mo...@gmail.com wrote: On 23 March 2014 00:00, Nick Coghlan ncogh...@gmail.com wrote: On 23 Mar 2014 09:35, Paul Moore p.f.mo...@gmail.com wrote: On 22 March 2014 22:37, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: 1. Formally decouple the setup.py CLI from the distutils command system Agreed - it looks like a compatible CLI needs to be part of the transition to any new system (I assume that's what you meant). Does this statement mean document the exact setup.py invocations which need to be supported by any tool or does it mean create a new standard cli and update pip to use it and sort out a means of wrapping existing setup.py scripts to expose it? If the former, then I've informally done that earlier in the thread (basically setup.py bdist_wheel -d XXX is it[1], if we assume that by the time it's official pip will have completed the move to installing from sdist by using wheels as an intermediate step). Writing a wrapper setup.py that simply invokes a different tool such as bento is trivial. I mean actually writing it up and documenting all the *build options* (including cross compilation support). As far as I am aware, that hasn't been done to date (at least, not in an easily consumable form that people can link to). Hmm. How about if, to start the ball rolling, I add a short section to the pip documentation entitled setup.py interface that explains the basic commands that pip uses. That can then be expanded on as additional data is collected. Yes, explaining the interface that pip currently relies on would definitely be a good starting point. One thing though. I may be being dense here, but I'm not aware of any way for pip to provide build options via pip wheel. I certainly don't know the pip wheel command you'd use to (for example) cross-compile psutil for Windows on a Linux box. Or how you'd use pip wheel to build pyYAML with libyaml support (you need to indicate where the libyaml headers and library live). Obviously you can do this with *distutils*, but that's different. We're not expecting bento, or distil, or waf, or whatever, to support the full distutils command line interface, surely? Now you begin to see the scope of the problem. It's definitely solvable, but means asking a whole pile of required, recommended or distutils-specific? questions about the existing distutils and setuptools build system :) pip already relies on it sets the minimum for the required category, but there's more to a full build system abstraction than what pip currently supports. Cheers, Nick. The pip wheel command includes --build-options and --build-options flags that translate to global and command-specific options on the bdist_wheel command. Those are as follows: Global options: --verbose (-v) run verbosely (default) --quiet (-q)run quietly (turns verbosity off) --dry-run (-n) don't actually do anything --help (-h) show detailed help message --no-user-cfg ignore pydistutils.cfg in your home directory Options for 'bdist_wheel' command: --bdist-dir (-b) temporary directory for creating the distribution --plat-name (-p) platform name to embed in generated filenames (default: win-amd64) --keep-temp (-k) keep the pseudo-installation tree around after creating the distribution archive --dist-dir (-d) directory to put final built distributions in --skip-build skip rebuilding everything (for testing/debugging) --relativebuild the archive using relative paths(default: false) --owner (-u) Owner name used when creating a tar file [default: current user] --group (-g) Group name used when creating a tar file [default: current group] --skip-scriptsskip building the setuptools console_scripts --universal make a universal wheel (default: false) Note that none of those control the build - the ones that do that are options to the build_ext and build commands, but *distutils* doesn't allow you to supply them in a bdist_wheel command. The same is true for pip install, BTW. That sends the options to the setup.py install command - again, not to setup.py build_ext. I can imagine good ways of using the existing pip interface to drive build tools (even distutils!) more effectively than we do at present. But I don't think the current interface supports compilation arguments (even if it looks like it does). Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On 23 March 2014 23:13, Nick Coghlan ncogh...@gmail.com wrote: Now you begin to see the scope of the problem. It's definitely solvable, but means asking a whole pile of required, recommended or distutils-specific? questions about the existing distutils and setuptools build system :) pip already relies on it sets the minimum for the required category, but there's more to a full build system abstraction than what pip currently supports. OK, I see now. So the ultimate build system will include pip changes to supply build-time options in an as-yet unspecified manner. There's certainly no way I can do all of that myself, I don't have remotely the level of experience with complex build requirements. But I can probably take the first steps, and leave it to people with the experience to add to it. No promises on timescales but I'll see what I can do. One thought. do we want to use a setup.py script as the interface, with all its historical baggage, or would we be better using a new script name as the official interface (with pip falling back to equivalent setup.py invocations when that script isn't present, for backward compatibility)? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
1. Formally decouple the setup.py CLI from the distutils command system Agreed - it looks like a compatible CLI needs to be part of the transition to any new system (I assume that's what you meant). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On 23 March 2014 08:37, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: 1. Formally decouple the setup.py CLI from the distutils command system Agreed - it looks like a compatible CLI needs to be part of the transition to any new system (I assume that's what you meant). Yeah - we have precedent there with distutils2, where they needed d2to1 to allow it to be used with easy_install and pip, and I think that model makes sense. The main blockers at the moment are to get the setuptools CLI docs up to date, and then to run through that and nominate what represents the core functionality that can be expected from a reimplementation, and what can be deemed setuptools specific. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On 22 March 2014 22:37, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: 1. Formally decouple the setup.py CLI from the distutils command system Agreed - it looks like a compatible CLI needs to be part of the transition to any new system (I assume that's what you meant). Does this statement mean document the exact setup.py invocations which need to be supported by any tool or does it mean create a new standard cli and update pip to use it and sort out a means of wrapping existing setup.py scripts to expose it? If the former, then I've informally done that earlier in the thread (basically setup.py bdist_wheel -d XXX is it[1], if we assume that by the time it's official pip will have completed the move to installing from sdist by using wheels as an intermediate step). Writing a wrapper setup.py that simply invokes a different tool such as bento is trivial. Paul [1] The one exception is editable installations (setup.py develop). That's supported by pip, implemented wholly in setuptools, and an integral part of many workflows. I don't believe anyone has looked yet at what might be needed to decouple that from the distutils/setuptools/setup.py infrastructure. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On 23 Mar 2014 09:35, Paul Moore p.f.mo...@gmail.com wrote: On 22 March 2014 22:37, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: 1. Formally decouple the setup.py CLI from the distutils command system Agreed - it looks like a compatible CLI needs to be part of the transition to any new system (I assume that's what you meant). Does this statement mean document the exact setup.py invocations which need to be supported by any tool or does it mean create a new standard cli and update pip to use it and sort out a means of wrapping existing setup.py scripts to expose it? If the former, then I've informally done that earlier in the thread (basically setup.py bdist_wheel -d XXX is it[1], if we assume that by the time it's official pip will have completed the move to installing from sdist by using wheels as an intermediate step). Writing a wrapper setup.py that simply invokes a different tool such as bento is trivial. I mean actually writing it up and documenting all the *build options* (including cross compilation support). As far as I am aware, that hasn't been done to date (at least, not in an easily consumable form that people can link to). Cheers, Nick. Paul [1] The one exception is editable installations (setup.py develop). That's supported by pip, implemented wholly in setuptools, and an integral part of many workflows. I don't believe anyone has looked yet at what might be needed to decouple that from the distutils/setuptools/setup.py infrastructure. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
Bento emulates setup.py: https://github.com/cournape/Bento/blob/master/setup.py It mostly has to implement egg_info and install to be pip compatible. In the future we'd like them to implement dist_info (generate a .dist-info directory instead of an .egg-info directory) and wheel; an sdist would signal that it doesn't implement setup.py install and pip would always build and then install a wheel. This would be a relatively simple thing to add to pip. Once setup.py emulation is working then we could define a Python-level plugin interface for build systems that might return metadata dicts, and also provide a setup.py that talks to this plugin interface. A later version of pip might talk directly to the plugin without bothering with setup.py. This strategy does not generally try to eliminate arbitrary code execution during builds - builds are an inherently arbitrary-code process. But once the build has happened most installs should work without arbitrary code execution. On Fri, Mar 21, 2014 at 6:13 AM, Nick Coghlan ncogh...@gmail.com wrote: On 20 Mar 2014 23:16, Brett Cannon bcan...@gmail.com wrote: On Thu Mar 20 2014 at 12:51:13 AM, Nick Coghlan ncogh...@gmail.com wrote: On 20 March 2014 09:54, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Daniel Holth dholth at gmail.com writes: extensions without using distutils. The problem of invoking the compiler has been solved by many build systems and is not just a unique and mysterious distutils feature. Did someone say it was? Building extensions is something distil does too, and without using distutils or setuptools. Right, the problem is the lack of a standard interface for how the packaging system is supposed to invoke them - that is, *implementation independent* docs of what the various setup.py commands are *supposed* to do. The packaging system shouldn't have to care *how* setup.py is implemented, but at the moment, the behaviour is underspecified to the point where it's a matter of reverse engineering distutils and/or implementing what seems necessary and waiting to see if people complain. What are the plans for the build step in the grand plan of Python packaging? I think previously it has been suggested that once metadata is done and distribution/installation is taken care of the distutils/setuptools building part of all of this will be tackled. Is that still accurate? That's the priority list for my personal focus, but I certainly wouldn't complain if there was momentum behind addressing the build side in parallel, rather than in series. Commenting on PEPs is usually easier than writing them, and the starting point for the build side is mostly just defining a suitable subset of setup.py commands, options, behaviour and invocation context that will make up the fallback legacy interface a new metabuild system would need regardless of any other details. I just don't have the personal bandwidth to champion such an effort at this point in time. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On 21 March 2014 13:46, Daniel Holth dho...@gmail.com wrote: It mostly has to implement egg_info and install to be pip compatible. At a *very* brief scan, pip uses just the following setup.py invocations (this definitely needs to be properly audited before relying on it): For installation from sdist * setup.py egg_info [--egg-base XXX] * setup.py install --record XXX [--single-version-externally-managed] [--root XXX] [--compile|--no-compile] [--install-headers XXX] For editable installs * setup.py develop --no-deps Not sure, used in RequirementSet * setup.py clean For pip wheel * setup.py bdist_wheel -d XXX Note that installation from wheel requires *no* setup.py interface. Nearly all of this is for installation from sdist (which long-term is expected to be deprecated - install via wheels is the intended way of the future, and PR #1572 is the first step on the way to that). While theoretically you can build wheels any way you like, I expect pip wheel XXX to be the canonical approach (it's how PR#1572 works) and at the moment that means implementing setup.py bdist_wheel. So, at a bare minimum, you could get away with just supporting setup.py bdist_wheel (and not supporting editable installs or installs direct from source). It might need PR #1572 to make this palatable, of course. Also, there's the fact (which I glossed over) that pip currently doesn't run setup.py directly, but does some nasty hackery to inject setuptools. Before working on standard build interfaces, we probably need to refactor that out somehow. Maybe a simple flag (that could be set in setup.cfg) that says use setup.py without monkeypatching would be enough... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On Fri, Mar 21, 2014 at 10:25 AM, Paul Moore p.f.mo...@gmail.com wrote: On 21 March 2014 13:46, Daniel Holth dho...@gmail.com wrote: It mostly has to implement egg_info and install to be pip compatible. At a *very* brief scan, pip uses just the following setup.py invocations (this definitely needs to be properly audited before relying on it): For installation from sdist * setup.py egg_info [--egg-base XXX] * setup.py install --record XXX [--single-version-externally-managed] [--root XXX] [--compile|--no-compile] [--install-headers XXX] For editable installs * setup.py develop --no-deps Not sure, used in RequirementSet * setup.py clean For pip wheel * setup.py bdist_wheel -d XXX Note that installation from wheel requires *no* setup.py interface. Nearly all of this is for installation from sdist (which long-term is expected to be deprecated - install via wheels is the intended way of the future, and PR #1572 is the first step on the way to that). While theoretically you can build wheels any way you like, I expect pip wheel XXX to be the canonical approach (it's how PR#1572 works) and at the moment that means implementing setup.py bdist_wheel. So, at a bare minimum, you could get away with just supporting setup.py bdist_wheel (and not supporting editable installs or installs direct from source). It might need PR #1572 to make this palatable, of course. Also, there's the fact (which I glossed over) that pip currently doesn't run setup.py directly, but does some nasty hackery to inject setuptools. Before working on standard build interfaces, we probably need to refactor that out somehow. Maybe a simple flag (that could be set in setup.cfg) that says use setup.py without monkeypatching would be enough... No worries... if you are re-implementing setup.py, then setuptools is monkey-patching code that you won't be executing. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On Fri, Mar 21, 2014 at 10:25 AM, Paul Moore p.f.mo...@gmail.com wrote: On 21 March 2014 13:46, Daniel Holth dho...@gmail.com wrote: It mostly has to implement egg_info and install to be pip compatible. At a *very* brief scan, pip uses just the following setup.py invocations (this definitely needs to be properly audited before relying on it): For installation from sdist * setup.py egg_info [--egg-base XXX] * setup.py install --record XXX [--single-version-externally-managed] [--root XXX] [--compile|--no-compile] [--install-headers XXX] For editable installs * setup.py develop --no-deps Not sure, used in RequirementSet * setup.py clean For pip wheel * setup.py bdist_wheel -d XXX Note that installation from wheel requires *no* setup.py interface. Nearly all of this is for installation from sdist (which long-term is expected to be deprecated - install via wheels is the intended way of the future, and PR #1572 is the first step on the way to that). While theoretically you can build wheels any way you like, I expect pip wheel XXX to be the canonical approach (it's how PR#1572 works) and at the moment that means implementing setup.py bdist_wheel. So, at a bare minimum, you could get away with just supporting setup.py bdist_wheel (and not supporting editable installs or installs direct from source). It might need PR #1572 to make this palatable, of course. You do need to keep a mechanism to indicate install build dependencies. Also, there's the fact (which I glossed over) that pip currently doesn't run setup.py directly, but does some nasty hackery to inject setuptools. Before working on standard build interfaces, we probably need to refactor that out somehow. Maybe a simple flag (that could be set in setup.cfg) that says use setup.py without monkeypatching would be enough... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On 21 March 2014 17:56, Daniel Holth dho...@gmail.com wrote: You do need to keep a mechanism to indicate install build dependencies. Do we have that at the moment? Metadata 2.0 covers that with build_requires. At the moment setup_requires covers it, but I don't know if/how that works (AIUI, setup_requires is processed by setuptools, so it's somewhat outside of the normal pip mechanisms). But yes, there needs to be some accessible data that tells pip install these packages before trying to run setup.py. Again, though, that's only needed for pip wheel or for the install-direct-from-source cases that may not even be expected to work in the new metabuild system... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On Fri, Mar 21, 2014 at 2:37 PM, Paul Moore p.f.mo...@gmail.com wrote: On 21 March 2014 17:56, Daniel Holth dho...@gmail.com wrote: You do need to keep a mechanism to indicate install build dependencies. Do we have that at the moment? Metadata 2.0 covers that with build_requires. At the moment setup_requires covers it, but I don't know if/how that works (AIUI, setup_requires is processed by setuptools, so it's somewhat outside of the normal pip mechanisms). But yes, there needs to be some accessible data that tells pip install these packages before trying to run setup.py. Again, though, that's only needed for pip wheel or for the install-direct-from-source cases that may not even be expected to work in the new metabuild system... Paul We will always need the Gentoo-model compiled-on-demand style of installation. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/21/2014 02:37 PM, Paul Moore wrote: Again, though, that's only needed for pip wheel or for the install-direct-from-source cases that may not even be expected to work in the new metabuild system... Hmm, that is a clear case of baby-with-the-bathwater if I ever saw one. Installing from an sdist has got to be orders-of-magnitude more common than anything requiring a separate compilation / build step. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMsieIACgkQ+gerLs4ltQ4cTACfbzcmVCMkAcXw8/4IWmNonAZV hlYAn2iNppd+ICEz7L2qr4sxnIISqykY =Dzaq -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On Mar 21, 2014, at 2:50 PM, Tres Seaver tsea...@palladion.com wrote: Signed PGP part On 03/21/2014 02:37 PM, Paul Moore wrote: Again, though, that's only needed for pip wheel or for the install-direct-from-source cases that may not even be expected to work in the new metabuild system... Hmm, that is a clear case of baby-with-the-bathwater if I ever saw one. Installing from an sdist has got to be orders-of-magnitude more common than anything requiring a separate compilation / build step. Tres. -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig Some confusion here :) When pip switches to only install from Wheels it’ll still handle sdists, it’ll just do them by invoking the metabuild system to create a Wheel and then install that Wheel. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On Fri, Mar 21, 2014 at 1:46 PM, Daniel Holth dho...@gmail.com wrote: Bento emulates setup.py: https://github.com/cournape/Bento/blob/master/setup.py It mostly has to implement egg_info and install to be pip compatible. Yes. While some things are quite not ideal in Bento, I think that part is a good one: build a system that goes away completely from distutils, and then emulate those interfaces (where it makes sense). This allows compatibility in most cases, while avoiding re-introducing what makes distutils horrible (commands and compilers). Before attempting to solve the build problem itself, I would suggest to have a small library extracted from distutils that does not impose any 'policy' on the build, but expose compilation information (flags, link options, etc...). This library could be used from distutils, but also (crucially) by 3rd party tools. Regarding build, my own experience after having dealt with unholly scons/waf/distutils/numpy.distutils integration projects: - solving the simple cases (simple C extensions) is easy, but moving to more complex cases makes the problem much harder. Build is really a case where solving 80 % of the usecases is solving 10 % of the problem. Solving 80 % of the usecases is ok as long as it does not prevent the remaining 20 % to do what they want. - neither scons or waf are really usable as libraries (scons much more so than waf). Even fbuild, which I generally find more approachable, has a 'god' object that makes integration difficult. - having well documented, self-contained API to build final artefacts (sdist, installs, eggs, wheels, bdist_wininst, bdist_msi, etc...) that help integrating with 3rd party build tools is more critical than the build tool itself. David In the future we'd like them to implement dist_info (generate a .dist-info directory instead of an .egg-info directory) and wheel; an sdist would signal that it doesn't implement setup.py install and pip would always build and then install a wheel. This would be a relatively simple thing to add to pip. Once setup.py emulation is working then we could define a Python-level plugin interface for build systems that might return metadata dicts, and also provide a setup.py that talks to this plugin interface. A later version of pip might talk directly to the plugin without bothering with setup.py. This strategy does not generally try to eliminate arbitrary code execution during builds - builds are an inherently arbitrary-code process. But once the build has happened most installs should work without arbitrary code execution. On Fri, Mar 21, 2014 at 6:13 AM, Nick Coghlan ncogh...@gmail.com wrote: On 20 Mar 2014 23:16, Brett Cannon bcan...@gmail.com wrote: On Thu Mar 20 2014 at 12:51:13 AM, Nick Coghlan ncogh...@gmail.com wrote: On 20 March 2014 09:54, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Daniel Holth dholth at gmail.com writes: extensions without using distutils. The problem of invoking the compiler has been solved by many build systems and is not just a unique and mysterious distutils feature. Did someone say it was? Building extensions is something distil does too, and without using distutils or setuptools. Right, the problem is the lack of a standard interface for how the packaging system is supposed to invoke them - that is, *implementation independent* docs of what the various setup.py commands are *supposed* to do. The packaging system shouldn't have to care *how* setup.py is implemented, but at the moment, the behaviour is underspecified to the point where it's a matter of reverse engineering distutils and/or implementing what seems necessary and waiting to see if people complain. What are the plans for the build step in the grand plan of Python packaging? I think previously it has been suggested that once metadata is done and distribution/installation is taken care of the distutils/setuptools building part of all of this will be tackled. Is that still accurate? That's the priority list for my personal focus, but I certainly wouldn't complain if there was momentum behind addressing the build side in parallel, rather than in series. Commenting on PEPs is usually easier than writing them, and the starting point for the build side is mostly just defining a suitable subset of setup.py commands, options, behaviour and invocation context that will make up the fallback legacy interface a new metabuild system would need regardless of any other details. I just don't have the personal bandwidth to champion such an effort at this point in time. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org
Re: [Distutils] waf can be used to compile Python extension modules
This strategy does not generally try to eliminate arbitrary code execution during builds - builds are an inherently arbitrary-code process. But once the build has happened most installs should work without arbitrary code execution. I don't think builds should be a *completely* arbitrary-code process. I understand well that user-defined code should be accommodated, but IMO this should be within a declarative framework with well-defined hooks, otherwise it will ultimately lead to the same problems that setup.py has. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On 22 March 2014 09:37, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: This strategy does not generally try to eliminate arbitrary code execution during builds - builds are an inherently arbitrary-code process. But once the build has happened most installs should work without arbitrary code execution. I don't think builds should be a *completely* arbitrary-code process. I understand well that user-defined code should be accommodated, but IMO this should be within a declarative framework with well-defined hooks, otherwise it will ultimately lead to the same problems that setup.py has. Agreed, but that can be a two step process: 1. Formally decouple the setup.py CLI from the distutils command system 2. Define a more declarative metabuild system, with the setup.py CLI considered a legacy compatibility interface. The reason I think we need to cover step 1 first is because without doing that, I don't believe we're going to understand the existing scope we need to cover for 2. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
I don't think there is a need to be that pessimistic. Most people will even be able to keep setup.py as is. But when you really should be sidestepping DistUtils instead of writing a 10k line extension there is a supported path. On Mar 21, 2014 7:37 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: This strategy does not generally try to eliminate arbitrary code execution during builds - builds are an inherently arbitrary-code process. But once the build has happened most installs should work without arbitrary code execution. I don't think builds should be a *completely* arbitrary-code process. I understand well that user-defined code should be accommodated, but IMO this should be within a declarative framework with well-defined hooks, otherwise it will ultimately lead to the same problems that setup.py has. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On Thu Mar 20 2014 at 12:51:13 AM, Nick Coghlan ncogh...@gmail.com wrote: On 20 March 2014 09:54, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Daniel Holth dholth at gmail.com writes: extensions without using distutils. The problem of invoking the compiler has been solved by many build systems and is not just a unique and mysterious distutils feature. Did someone say it was? Building extensions is something distil does too, and without using distutils or setuptools. Right, the problem is the lack of a standard interface for how the packaging system is supposed to invoke them - that is, *implementation independent* docs of what the various setup.py commands are *supposed* to do. The packaging system shouldn't have to care *how* setup.py is implemented, but at the moment, the behaviour is underspecified to the point where it's a matter of reverse engineering distutils and/or implementing what seems necessary and waiting to see if people complain. What are the plans for the build step in the grand plan of Python packaging? I think previously it has been suggested that once metadata is done and distribution/installation is taken care of the distutils/setuptools building part of all of this will be tackled. Is that still accurate? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] waf can be used to compile Python extension modules
One reason people use distutils is because it can compile Python extension modules. Here's an example from waf that compiles Python extensions without using distutils. The problem of invoking the compiler has been solved by many build systems and is not just a unique and mysterious distutils feature. https://code.google.com/p/waf/source/browse/#git%2Fdemos%2Fpython ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
The idea is not new. Here are some examples for SCons: https://stackoverflow.com/questions/2627731/using-scons-as-a-build-engine-for-distutils The answers may need to be updated, so if anybody comes up with convenient distutils tool semantics - it can be integrated into next SCons release. On Wed, Mar 19, 2014 at 5:53 PM, Daniel Holth dho...@gmail.com wrote: One reason people use distutils is because it can compile Python extension modules. Here's an example from waf that compiles Python extensions without using distutils. The problem of invoking the compiler has been solved by many build systems and is not just a unique and mysterious distutils feature. https://code.google.com/p/waf/source/browse/#git%2Fdemos%2Fpython ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig -- anatoly t. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
Daniel Holth dholth at gmail.com writes: extensions without using distutils. The problem of invoking the compiler has been solved by many build systems and is not just a unique and mysterious distutils feature. Did someone say it was? Building extensions is something distil does too, and without using distutils or setuptools. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On 20 March 2014 09:54, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Daniel Holth dholth at gmail.com writes: extensions without using distutils. The problem of invoking the compiler has been solved by many build systems and is not just a unique and mysterious distutils feature. Did someone say it was? Building extensions is something distil does too, and without using distutils or setuptools. Right, the problem is the lack of a standard interface for how the packaging system is supposed to invoke them - that is, *implementation independent* docs of what the various setup.py commands are *supposed* to do. The packaging system shouldn't have to care *how* setup.py is implemented, but at the moment, the behaviour is underspecified to the point where it's a matter of reverse engineering distutils and/or implementing what seems necessary and waiting to see if people complain. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
