Re: [Distutils] People want CPAN
On Sun, Nov 15, 2009 at 10:33 PM, Greg Ewing wrote: > Tarek Ziadé wrote: > >> But the result is similar, and explicit imports should work too, so >> maybe registeries >> are just sugar on the top of something we first need to make work. > > It's completely unnecessary sugar, if you ask me. > I don't see what's bad about importing functionality > you want to use. > > Where and how do you intend the registration to happen, > anyway? Would it be done by the setup.py script? In > that case I don't see how it saves you anything, since > you would have to first import the thing you want to > register anyway. > > Or are you envisaging that Pyrex or whatever tool is > involved would somehow patch itself into distutils > when you install it? I don't like that idea much, > since it smacks of the kind of monkeypatching that > setuptools is reviled for. > Patching ? No, I was thinking about a basic plugin registery, exactly like what we have *now* for commands with distutils.cfg, which is a simple configparser file where you can point packages that contains commands, so they are loaded when Distutils is run. (that's the "command-packages" option) So, using the same technique, we can explicitely list in such .cfg what are the compilers and where they are: [compilers] pyrex=pyrex.distutils:PyrexCompile Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Tarek Ziadé wrote: But the result is similar, and explicit imports should work too, so maybe registeries are just sugar on the top of something we first need to make work. It's completely unnecessary sugar, if you ask me. I don't see what's bad about importing functionality you want to use. Where and how do you intend the registration to happen, anyway? Would it be done by the setup.py script? In that case I don't see how it saves you anything, since you would have to first import the thing you want to register anyway. Or are you envisaging that Pyrex or whatever tool is involved would somehow patch itself into distutils when you install it? I don't like that idea much, since it smacks of the kind of monkeypatching that setuptools is reviled for. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Sun, Nov 15, 2009 at 7:24 AM, Greg Ewing wrote:
> Tarek Ziadé wrote:
>
>> And have the community create new ExtensionBuilder subclasses that
>> could be registered like command.
>
> I don't see a need for registering anything. You should
> just be able to explicitly say what tool to use for each
> stage of the process.
>
> I envisage something like this:
>
> from distutils import Extension, CCompile
> from pyrex.distutils import PyrexCompile
>
> foo_ext = Extension("foo",
> CCompile(
> PyrexCompile("foo.pyx"),
> "somelib.c"))
>
> Here Extension, CCompile and PyrexCompile are constructors
> for dependency graph nodes. Their responsibilities are:
>
> Extension -- takes compiled object files and libraries
> and links them into a Python extension.
>
> CCompile -- takes C source files and turns them into
> object files.
>
> PyrexCompile -- takes Pyrex source files and turns
> them into C source files.
>
> They would of course also take other relevant arguments
> such as compiler flags, search paths, etc.
The advantage of the registery is that a project can provide a compiler type,
let's say "Pyrex". Then you can use in your own project setup.py this compiler
without explicitely importing something.
But the result is similar, and explicit imports should work too, so
maybe registeries
are just sugar on the top of something we first need to make work.
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Sun, Nov 15, 2009 at 3:44 AM, Jeremy Kloth wrote: [..] >> So did you end up changing the way options are passed to the commands, >> or do you just have a specific "config" command that looks over other >> options passed to the other commands ? > > It is done with the 'config' command having all the options used by the other > commands. The other commands would then look up their options' values from > 'config' (if not supplied on the command-line). If, for example, `--prefix` > was supplied to 'install', the 'install' command would then cause the 'config' > to redo its stored options. > > If at all possible, I would eliminate the redundant options on > build_*/install_* and leave them solely on 'config' as it greatly simplifies > the interaction of the commands. So, pratically speaking, if: $ python setup.py install is called, the install command will instanciate the configure command, that will return options that were stored in some file, in a previous call ? But, for the option redundancy problem, the simplest way I can see to have the "configure" options in "install", or to let the end user pass them along, would be to make "configure" the base class for all commands that are part of the configure-make-install story, so when they run they can read and write options in the stored file and use if needed the options passed in the command line. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Tarek Ziadé wrote:
And have the community create new ExtensionBuilder subclasses that
could be registered like command.
I don't see a need for registering anything. You should
just be able to explicitly say what tool to use for each
stage of the process.
I envisage something like this:
from distutils import Extension, CCompile
from pyrex.distutils import PyrexCompile
foo_ext = Extension("foo",
CCompile(
PyrexCompile("foo.pyx"),
"somelib.c"))
Here Extension, CCompile and PyrexCompile are constructors
for dependency graph nodes. Their responsibilities are:
Extension -- takes compiled object files and libraries
and links them into a Python extension.
CCompile -- takes C source files and turns them into
object files.
PyrexCompile -- takes Pyrex source files and turns
them into C source files.
They would of course also take other relevant arguments
such as compiler flags, search paths, etc.
--
Greg
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Saturday 14 November 2009 05:14:05 pm Tarek Ziadé wrote: > On Fri, Nov 13, 2009 at 3:46 PM, Jeremy Kloth > > The way we've impl'd 'config' was as a prerequisite for 'build', just as > > 'build' is for 'install'. If any of the options stored by the 'config' > > command are overridden via 'build' or 'install' options, the 'config' > > command would be re-run to store the "new" choices. > > > > Any questions or just simple help with integrating a similar system, just > > let me know. Been there, done that. > > So did you end up changing the way options are passed to the commands, > or do you just have a specific "config" command that looks over other > options passed to the other commands ? It is done with the 'config' command having all the options used by the other commands. The other commands would then look up their options' values from 'config' (if not supplied on the command-line). If, for example, `--prefix` was supplied to 'install', the 'install' command would then cause the 'config' to redo its stored options. If at all possible, I would eliminate the redundant options on build_*/install_* and leave them solely on 'config' as it greatly simplifies the interaction of the commands. Jeremy -- Jeremy Kloth http://4suite.org/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 11:49 PM, Pauli Virtanen wrote: [..] > > Just to throw some wild, perhaps obvious, and definitely unasked-for > ideas in the air (especially as I can't promise I can give any sustained > help here :/ ): > > I suppose one option would be to factor *everything* related to > extension module building into build_ext (and abolish build_clib): the > rest of distutils would just say to build_ext > > """ > Here's some info about the environment: > > dict(python_lib_path=/usr/lib/..., optimize=yes/no, > python_lib_name=..., python_includes=/usr/include/..., > install_prefix=/usr/lib/python2.6, ... > ..., python_extension=..., build_temp_dir=...) > > Look the rest up from sysconfig. > > Please build any extensions and data files you like, > and tell the file and directory names where you placed them > and where (relative paths) they should go. > """ I ought to be something like that, but what is unclear to me is how to describe which compiler to use for what files. I had this "one extension == one compiler type" pattern in my head, but it seems more complex than that. IOW an extension can invoke several compilers and tools to be built. So "one extension == one extension builder" might best describe it. and I am wondering if we can't define a simple interface for these extension builders, from the simplest case (one tool uses one compiler) to the weirdest one (one tool uses a complex toolchain to create the extension) So at the end we would have: - Extension (the existing extension class, that takes a "extension_builder_type") - ExtensionBuilder (class in charge of creating an extension) - a registery for ExtensionBuilder subclasses And have the community create new ExtensionBuilder subclasses that could be registered like command. build_ext would then become an empty shell, just in charge of looping through the extensions, so each extension invokes its builder. [..] > I think this idea quickly boils down more or less to David's idea about > a pluggable build system -- implementing a good one takes a lot of work, > so it might make sense to refactor distutils so that it would be > possible [1] to use some of the existing ones (scons, waf, whatever, > heck, even autoconf+make). The *default* build system could be a simple > one, and backwards compatible. Especially so, since it seems to me that > building extension modules is orthogonal to what distutils and > setuptools want to do -- package and install Python modules in a > Python-specific way, not really worry about how to properly call > different compilers on obscure platforms. > > Anyway, even in the case pluggability is a bad idea, refactoring the > build system out from the rest of distutils might make sense. Agreed. it seems that the addition of the "configure" command, and the refactoring of the "build_ext" one, are the right things to do, together with the addition of the "sysconfig" stdlib module (which allows configure to get more info that what distutils.sysconfig provides) Now, if we can take back the work done in 4suite as suggested, or in scons, etc, that's even better. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 3:46 PM, Jeremy Kloth wrote: [..] > > The exact thing being described has been done in 4Suite for 6 years (along > with many other distutils improvements). Feel free to take or discuss or > request help with any of the features/additions (like FHS layout of files) in > the 4Suite distutils extensions. The code is available for browsing: > http://cvs.4suite.org/viewcvs/4Suite/Ft/Lib/DistExt/ That's great, thanks for the pointer ! I'll look into it asap > > The way we've impl'd 'config' was as a prerequisite for 'build', just as > 'build' is for 'install'. If any of the options stored by the 'config' > command are overridden via 'build' or 'install' options, the 'config' command > would be re-run to store the "new" choices. > > Any questions or just simple help with integrating a similar system, just let > me know. Been there, done that. So did you end up changing the way options are passed to the commands, or do you just have a specific "config" command that looks over other options passed to the other commands ? Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
At 03:13 PM 11/11/2009 +0100, Tarek Ziadé wrote: But you call it with "install" in your example, meaning that is is called at install time, right ? Or it is just that you want to get the "--prefix" value finalized and computed by the install command. If it's the later, I guess you will be able to use the upcoming "sysconfig" module, that gives you the install schemes, depending on sys.prefix/sys.exec_prefix. The issue is that setup.py can accept multiple commands on the command line, and in principle "build_clib" might be being called as a subcommand of build (and thus of install). So, he needs the *active* --prefix, either from the command line, config file(s), or defaults. Simply having an API to get the defaults won't help this. Really, getting a finalized "install" command object is the only way to do this correctly in distutils at the moment, and the sysconfig API won't change that. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
ke, 2009-11-11 kello 21:42 +0100, Tarek Ziadé kirjoitti: [clip] > Do you mean, an Extension that would require several compilers ? > > I was thinking of a one-to-one relation between an Extension and a > compiler type, even if there are are multiple source files (in different > languages) > for this extension. > > Meaning that *one* compiler would have to handle of those files. Does that fit > the cython use case? It might. The distutils Cython compiler would however need to be smart enough to handle all languages used in the other source files by itself, or somehow delegate this back to distutils (which still sounds doable). Also, the f2py compiler does similar things. Ditto for Pyrex. How much code duplication would be needed between these? Source file preprocessing / generation is one thing that also comes useful. For example, Numpy has its own templated C code generator file format, and also generates some of the source files automatically from scratch during the build. These are compiled and linked as a part of ordinary C extension module. Currently, as I understand it, numpy.distutils does source generation in a separate build_src command. > > Also, the option of determining the necessary compiler and compiler > > options from file extensions does not always work. For example, Fortran > > 90 files come in two flavors, fixed-form and free-form, which often are > > not differentiated by the file extension. However, they may require > > different compiler flags to compile. (Hopefully, however, nobody uses > > the fixed-form for F90 any more...) > > How do they do then? Is file extensions, inline options in setup.py, > and the environ, are enough? numpy.distutils's Fortran compiler class reads the file and tries to analyze whether it's fixed-form or not. And chooses appropriate compiler flags. This is sort of a special "worst case" scenario for compiler selection, of course. *** Just to throw some wild, perhaps obvious, and definitely unasked-for ideas in the air (especially as I can't promise I can give any sustained help here :/ ): I suppose one option would be to factor *everything* related to extension module building into build_ext (and abolish build_clib): the rest of distutils would just say to build_ext """ Here's some info about the environment: dict(python_lib_path=/usr/lib/..., optimize=yes/no, python_lib_name=..., python_includes=/usr/include/..., install_prefix=/usr/lib/python2.6, ... ..., python_extension=..., build_temp_dir=...) Look the rest up from sysconfig. Please build any extensions and data files you like, and tell the file and directory names where you placed them and where (relative paths) they should go. """ Information needed how to build each component would be passed to build_ext subcomponent directly from setup.py or from a config file. Now, perhaps it is already like this --- I don't really know the internals of distutils --- and the build subcomponent is insulated from the others. In any case, something like this could make refactoring the build system easier. I think this idea quickly boils down more or less to David's idea about a pluggable build system -- implementing a good one takes a lot of work, so it might make sense to refactor distutils so that it would be possible [1] to use some of the existing ones (scons, waf, whatever, heck, even autoconf+make). The *default* build system could be a simple one, and backwards compatible. Especially so, since it seems to me that building extension modules is orthogonal to what distutils and setuptools want to do -- package and install Python modules in a Python-specific way, not really worry about how to properly call different compilers on obscure platforms. Anyway, even in the case pluggability is a bad idea, refactoring the build system out from the rest of distutils might make sense. .. [1] possible and easy -- I understand numpyscons is a stab at the possible, but it sounds like it was not easy to do. -- Pauli Virtanen ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Friday 13 November 2009 06:44:37 am Tarek Ziadé wrote: > Here's my proposal for this to happen, if you (and others) want to > contribute: > > Let's build this new "configure" command in Distribute 0.7, together > with the APIs to read/write the data. > > Then let's change the other commands consequently (without thinking > about backward compat first) > so we can try out this new behaviour. Fourthought's 4Suite distutils extensions already provide this behavior. You may consider looking at how the 'config' command is handled there as a guide in implementing this. Or just copy it wholesale, for that matter, as it is my doing. > Once it's proven to work good, we could publish Distribute 0.7 with > it, and depending on the community > feedback, we could integrate it to Distutils and work on the backward > compat part. > > this two phase step wouldn't be a problem imho, for early adopters > that would use and test it. The exact thing being described has been done in 4Suite for 6 years (along with many other distutils improvements). Feel free to take or discuss or request help with any of the features/additions (like FHS layout of files) in the 4Suite distutils extensions. The code is available for browsing: http://cvs.4suite.org/viewcvs/4Suite/Ft/Lib/DistExt/ The way we've impl'd 'config' was as a prerequisite for 'build', just as 'build' is for 'install'. If any of the options stored by the 'config' command are overridden via 'build' or 'install' options, the 'config' command would be re-run to store the "new" choices. Any questions or just simple help with integrating a similar system, just let me know. Been there, done that. Jeremy -- Jeremy Kloth http://4suite.org/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 1:52 PM, David Cournapeau wrote: > On Fri, Nov 13, 2009 at 9:46 PM, Tarek Ziadé wrote: >> On Fri, Nov 13, 2009 at 1:18 PM, Floris Bruynooghe >> wrote: >> [..] Is that scraping, or just preparing finalized options using "configure" ? Meaning other command would just have to get them when they run, if present ? How would that work ? configure would create a file ? >>> >>> That would be the obvious solution. Both autoconf and CPAN do this by >>> my understanding so it's a pretty logical thing to do. >> >> I find it logical too now. > > Scons and waf do it through a db kind of file, but that's because they > are much more fancy (scons for example keeps so called signature of > every command and nodes to know what has already been built or not). > > A first step as a plain file, which should clearly be marked as > implementation-defined and only guaranteed to work through an API is > the way to go I think. > Here's my proposal for this to happen, if you (and others) want to contribute: Let's build this new "configure" command in Distribute 0.7, together with the APIs to read/write the data. Then let's change the other commands consequently (without thinking about backward compat first) so we can try out this new behaviour. Once it's proven to work good, we could publish Distribute 0.7 with it, and depending on the community feedback, we could integrate it to Distutils and work on the backward compat part. this two phase step wouldn't be a problem imho, for early adopters that would use and test it. Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 9:46 PM, Tarek Ziadé wrote: > On Fri, Nov 13, 2009 at 1:18 PM, Floris Bruynooghe > wrote: > [..] >>> Is that scraping, or just preparing finalized options using "configure" ? >>> Meaning other command would just have to get them when they run, if present >>> ? >>> >>> How would that work ? configure would create a file ? >> >> That would be the obvious solution. Both autoconf and CPAN do this by >> my understanding so it's a pretty logical thing to do. > > I find it logical too now. Scons and waf do it through a db kind of file, but that's because they are much more fancy (scons for example keeps so called signature of every command and nodes to know what has already been built or not). A first step as a plain file, which should clearly be marked as implementation-defined and only guaranteed to work through an API is the way to go I think. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 1:18 PM, Floris Bruynooghe wrote: [..] >> Is that scraping, or just preparing finalized options using "configure" ? >> Meaning other command would just have to get them when they run, if present ? >> >> How would that work ? configure would create a file ? > > That would be the obvious solution. Both autoconf and CPAN do this by > my understanding so it's a pretty logical thing to do. I find it logical too now. [..] > Would it not be harder to add new command (or "build tasks") that > require information from the configure step in you structure it like > this? > I was thinking about an API that would allow any command to read/write the configuration data, and using it from the "configure" command to write it, and from the others to read it. That would allow including this new behaviour in existing commands with a deprecation step (in today's distutils, a build is triggered when you call install in any case, when there are some extensions) Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 11:26:23AM +0100, Tarek Ziadé wrote: > On Fri, Nov 13, 2009 at 6:22 AM, David Cournapeau > wrote: > > Tarek Ziadé wrote: > >> A deprecation warning would be added in install, if it finds a local > >> option, rather > >> than a global. Meaning both would work in 2.7/3.2. > >> > > > > If changing the command line in incompatible ways is acceptable, what do > > you think of scrapping the commands (at the UI level only) altogether ? > > This would be more consistent and easier to deal for the user, and > > easier to implement as well: > > > > python setup.py configure --option1 --option2=value2 > > python setup.py build > > python setup.py install > > > > We could then make this work: > > > > python setup.py install (would run both build and configure implicitly). > > Making all finalized options available at the build stage would then be > > easier. > > Is that scraping, or just preparing finalized options using "configure" ? > Meaning other command would just have to get them when they run, if present ? > > How would that work ? configure would create a file ? That would be the obvious solution. Both autoconf and CPAN do this by my understanding so it's a pretty logical thing to do. > It seems that you are pushing all the work in the "configure" option, > wich is fine to me, > but it also looks like you can already achieve this with the existing > system, by > changing the subcommands that are in the install command and their > order. That is: > > install > configure > build > all the install_* > > But if we want to see this working with "build" alone, configure has > to be a subcommand of build: > > install > build > configure > all the install_* Would it not be harder to add new command (or "build tasks") that require information from the configure step in you structure it like this? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 6:22 AM, David Cournapeau wrote: > Tarek Ziadé wrote: >> A deprecation warning would be added in install, if it finds a local >> option, rather >> than a global. Meaning both would work in 2.7/3.2. >> > > If changing the command line in incompatible ways is acceptable, what do > you think of scrapping the commands (at the UI level only) altogether ? > This would be more consistent and easier to deal for the user, and > easier to implement as well: > > python setup.py configure --option1 --option2=value2 > python setup.py build > python setup.py install > > We could then make this work: > > python setup.py install (would run both build and configure implicitly). > Making all finalized options available at the build stage would then be > easier. Is that scraping, or just preparing finalized options using "configure" ? Meaning other command would just have to get them when they run, if present ? How would that work ? configure would create a file ? It seems that you are pushing all the work in the "configure" option, wich is fine to me, but it also looks like you can already achieve this with the existing system, by changing the subcommands that are in the install command and their order. That is: install configure build all the install_* But if we want to see this working with "build" alone, configure has to be a subcommand of build: install build configure all the install_* IOW this would just require: 1/ adding a "configure" command 2/ inserting it as the first sub command of "build" 3/ make it possible, to share in the whole chain of commands, the passed arguments Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Tarek Ziadé wrote: > A deprecation warning would be added in install, if it finds a local > option, rather > than a global. Meaning both would work in 2.7/3.2. > If changing the command line in incompatible ways is acceptable, what do you think of scrapping the commands (at the UI level only) altogether ? This would be more consistent and easier to deal for the user, and easier to implement as well: python setup.py configure --option1 --option2=value2 python setup.py build python setup.py install We could then make this work: python setup.py install (would run both build and configure implicitly). Making all finalized options available at the build stage would then be easier. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Nov 12, 2009, at 12:42 PM, John Kleint wrote:
I have the start of such a document, derived from my own recent
experience pulling together all the information needed to release a
new Python project. It's aimed at people new to packaging and
possibly open source in general, so it's broader than just packaging.
It includes things like basic file layout, unit testing, Pylint,
getting started with Sphinx, a sample (distutils) setup.py, and how to
register with PyPI. I try to pull together all the info a novice
would need to have a reasonable shot at sharing their code. It's by
no means all-encompassing (e.g. no mention of native extensions), but
perhaps someone would be willing to write a complimentary advanced
packaging howto. You can find it at:
http://infinitemonkeycorps.net/docs/pph/
Let me know what you think.
It's a good start, but a few critques on the packaging section:
* Avoid importing your package before you've installed it.
Importing a package works fine for packages which have no
dependencies, but once your package depends upon other packages you
can't use this technique. It's better to teach people how to write
setup.py scripts which don't import the package so that their
setup.py's can be consistent once they do write packages with
dependencies. There are lots of cases where the assumptions of the
configuration of a developer's environment where dependencies are
installed that won't work for other people trying to install such a
package, which leads to setup.py files which don't work.
One example for handling the long_description field can be done
with (there are lots of variants on this, and you can just call open()
for a single text file:
def read(*rnames):
return open(os.path.join(os.path.dirname(__file__),
*rnames)).read()
long_description=read(
read('README.txt')
+ '\n' +
'Detailed Documentation\n'
'**\n'
+ '\n' +
'src/some-other-file.txt',
)
* download_url field is not necessary.
Libraries are often installed automatically by tools (buildout,
pip, easy_install). This work best if the downloads are in a
consistent structure (e.g. on PyPI). If releases are made to PyPI,
then you don't need a download_url to get to another place where the
file can be downloaded. The field is also prone to mistakes, since
easy_install will follow the download URL and screen-scrape that page
looking for a download link. Which can lead to older or wrong versions
of the package being downloaded. Basically it just causes room for
errors, and is just one extra step that you need to do to create a
package, so why do it?
* package_dir location.
This is a very minor point, but I find putting the source in a
directory named after the module or package which isn't the module or
package confusing. A common alternative is to use 'src' as the name
for the source directory. Then that location can be consistent from
package to package.
* provides (and requires) are scheduled for deprecation
These fields don't make any sense. If you are listing the imports
that your package needs, you need to list the name of the distribution
which provides those packages, not the names of the imports that your
code is using. setuptools (or now distribute) lets you use an
'install_requires' field, and this fields is also likely to be in
Distutils in the next Python release.
* Use a tool to help automate releases
Creating each release requires a lot of manual changes here and
there (bump the version, update the changelog, run sdist, upload to
pypi). In the spirit of "release early, release often", I like to use
a tool to automate releases, and also helps establish a few more
common conventions between Python projects. I'm using zest.releaser
for this ATM (shameless plug, since I'm also a contributor to the
project):
http://pypi.python.org/pypi/zest.releaser
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 1:08 AM, David Cournapeau wrote:
[..]
>
>> $ python setup.py --prefix=foo cmd1 cmd1 etc
>>
>> and the result would be in Distribution.options = {'path1': xxx, 'path2': xx}
>
> This is a major change in distutils behavior, so we need to solve the
> following issues:
> - every user will have to change how to call distutils
> - what happens if people still use python setup.py install
> --option1=foo instead of python setup.py --prefix=option1 install
A deprecation warning would be added in install, if it finds a local
option, rather
than a global. Meaning both would work in 2.7/3.2.
That would give them 18 to 24 months * 2 to go with the new style.
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 8:47 AM, Tarek Ziadé wrote:
>
> I am not sure to follow here: let's forget about your example where
> you call build_src and install together.
>
> -> in the real world, how a --prefix passed to install is going to
> impact a build command and vice-versa ?
In many cases. I gave the rpath example, but there is also the
pkg-config-like files generation example, etc... It is not possible to
foresee usage of this in a build tool, any sufficiently complex
project will need to share those options, and different ones depending
on the tool. I think it is a uncontroversial design decision followed
by almost every build tool.
> So, if you are not *installing*, it doesn't make sense to call the
> *install* command, and build could
> have its --prefix option in your case.
I don't *want* to call the install command, but I want to know the
prefix option of install. I do not want a build specific prefix
option, I want to know the global install option, whatever the user
command lines are.
prefix is only an example - as I mentioned in my previous email, I
potentially need every option of install in build command.
> $ python setup.py --prefix=foo cmd1 cmd1 etc
>
> and the result would be in Distribution.options = {'path1': xxx, 'path2': xx}
This is a major change in distutils behavior, so we need to solve the
following issues:
- every user will have to change how to call distutils
- what happens if people still use python setup.py install
--option1=foo instead of python setup.py --prefix=option1 install
cheers,
David
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Thu, Nov 12, 2009 at 11:45 PM, David Cournapeau wrote:
[..]
> I think I was confusing with my rpath example, which may be the source
> of our misunderstanding. I don't want to add a --prefix option to
> build_ext. I want to support the following user cases:
>
> python setup.py build # here, prefix would be whatever default, as
> available from sysconfig
> python setup.py install --prefix=foo # here, prefix would be the one
> as computer by install command
> python setup.py build_ext -i # here prefix is the current directory
>
> Requiring users to handle new options of commands is impractical IMHO,
> and a prefix option to build has a strange feeling to it.
I am not sure to follow here: let's forget about your example where
you call build_src and install together.
-> in the real world, how a --prefix passed to install is going to
impact a build command and vice-versa ?
install just copy files, where it's told to copy them, and build does
some work with some options,
and as a matter of fact you seem to use installation prefix at this stage.
So, if you are not *installing*, it doesn't make sense to call the
*install* command, and build could
have its --prefix option in your case.
>
>> Remember that the build step has nothing to do with the install step
>
> Ideally, what we want here is like autoconf does it. You have a
> configure step, and then at build/install stages, you have access to
> *all* options. Those options can be customized at build through usual
> make mechanism, but this is taken into account as well without the
> makefile writer having to care.
>
> IOW, we need to know *all* the finalized options *before* the first
> build command is even run.
So, IOW, these options cannot be finalized by *any* command. So the
proposal I've made
to have global options that are common to all commands, and that can
be used to create
an execution environment through public APIs would help in there.
So if the install command transforms "prefix" into "path1" "path2" and
"path3", we need this transformation
to occur outside install, so it can be done before the commands are run.
that is:
$ python setup.py --prefix=foo cmd1 cmd1 etc
and the result would be in Distribution.options = {'path1': xxx, 'path2': xx}
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Fri, Nov 13, 2009 at 7:12 AM, Tarek Ziadé wrote:
> On Thu, Nov 12, 2009 at 12:31 PM, Pauli Virtanen wrote:
>> Thu, 12 Nov 2009 10:54:41 +0100, Tarek Ziadé wrote:
>> [clip]
get_install_paths('FOO')
>
> And that's the API we want to add in sysconfig, roughly.
That does not solve the problem about getting FOO in the first place
when calling get_install_path.
>>>
>>> Why that ? where "FOO" comes from ? if it's an option you provide at
>>> build time like you said, earlier, you just pass it to the API to get
>>> the paths.
>>>
>>> FOO is not coming from nowhere...
>>
>> If this is not painfully clear already, the user passes FOO to the command
>>
>> python setup.py install --prefix=FOO
>>
>> Now, clearly the distutils install subcommand knows what FOO is.
>> But does build_ext? Does sysconfig?
>
> The install command takes FOO, and build several paths with it:
>
> /FOO/lib/site-package/
> /FOO/xxx
>
> The other commands that needs to get the same path can rebuild it using an
> API,
> that does:
>
> get_paths(scheme, vars={'prefix': 'FOO')
>
> Instead of doing what David did:
>
> $ python setup.py build_ext install --prefix=FOO
>
> They can do:
>
> $ python setup.py build_ext --prefix=FOO
>
> and don't require to use the install command anymore to get these paths
> cooked.
I think I was confusing with my rpath example, which may be the source
of our misunderstanding. I don't want to add a --prefix option to
build_ext. I want to support the following user cases:
python setup.py build # here, prefix would be whatever default, as
available from sysconfig
python setup.py install --prefix=foo # here, prefix would be the one
as computer by install command
python setup.py build_ext -i # here prefix is the current directory
Requiring users to handle new options of commands is impractical IMHO,
and a prefix option to build has a strange feeling to it.
> Remember that the build step has nothing to do with the install step
Ideally, what we want here is like autoconf does it. You have a
configure step, and then at build/install stages, you have access to
*all* options. Those options can be customized at build through usual
make mechanism, but this is taken into account as well without the
makefile writer having to care.
IOW, we need to know *all* the finalized options *before* the first
build command is even run.
David
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Thu, Nov 12, 2009 at 9:42 PM, John Kleint wrote: >>Tarek Ziadé schrieb: >> >>> For the documentation part I am afraid it will be messy for the end >>> users trying to package apps in Python *until* all PEPs >>> have made it into Python. >>> >>> Although, as Ian Bicking says: we could write today some kind of >>> all-in-one tutorial so end-users can work out without having to run >>> after the documenation in several places. >> >>Big +1, and I would be more than willing to include it in the standard >>Python documentation, *even* if it mostly describes setuptools/distribute/pip. >> >>When people want to package a library, they *will* look for docs in Python, >>and at the moment they only find the distutils reference. While the latter >>is necessary, a more howto-like standard document is much needed. >> >>Georg > > > I have the start of such a document, derived from my own recent > experience pulling together all the information needed to release a > new Python project. It's aimed at people new to packaging and > possibly open source in general, so it's broader than just packaging. > It includes things like basic file layout, unit testing, Pylint, > getting started with Sphinx, a sample (distutils) setup.py, and how to > register with PyPI. I try to pull together all the info a novice > would need to have a reasonable shot at sharing their code. It's by > no means all-encompassing (e.g. no mention of native extensions), but > perhaps someone would be willing to write a complimentary advanced > packaging howto. You can find it at: > > http://infinitemonkeycorps.net/docs/pph/ > > Let me know what you think. That's a great start indeed. If we can expand on the packaging thing we could take it from here and see who wants to help on this. I could try to convince some developers from pip and virtualenv to help on this. Are you willing to share it in a DVCS or something for that work ? How would you like to manage contributions in it ? Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Thu, Nov 12, 2009 at 12:31 PM, Pauli Virtanen wrote:
> Thu, 12 Nov 2009 10:54:41 +0100, Tarek Ziadé wrote:
> [clip]
>>> get_install_paths('FOO')
And that's the API we want to add in sysconfig, roughly.
>>>
>>> That does not solve the problem about getting FOO in the first place
>>> when calling get_install_path.
>>
>> Why that ? where "FOO" comes from ? if it's an option you provide at
>> build time like you said, earlier, you just pass it to the API to get
>> the paths.
>>
>> FOO is not coming from nowhere...
>
> If this is not painfully clear already, the user passes FOO to the command
>
> python setup.py install --prefix=FOO
>
> Now, clearly the distutils install subcommand knows what FOO is.
> But does build_ext? Does sysconfig?
The install command takes FOO, and build several paths with it:
/FOO/lib/site-package/
/FOO/xxx
The other commands that needs to get the same path can rebuild it using an API,
that does:
get_paths(scheme, vars={'prefix': 'FOO')
Instead of doing what David did:
$ python setup.py build_ext install --prefix=FOO
They can do:
$ python setup.py build_ext --prefix=FOO
and don't require to use the install command anymore to get these paths cooked.
Remember that the build step has nothing to do with the install step,
and David in his example was doing a build that was not relocatable,
e.g. receiving the install paths and applying them in the binaries.
So instead of having this paths created by code in the install command,
they are moved in an API than can be used by install or by any other command.
Is that clearer ?
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
>Tarek Ziadé schrieb: > >> For the documentation part I am afraid it will be messy for the end >> users trying to package apps in Python *until* all PEPs >> have made it into Python. >> >> Although, as Ian Bicking says: we could write today some kind of >> all-in-one tutorial so end-users can work out without having to run >> after the documenation in several places. > >Big +1, and I would be more than willing to include it in the standard >Python documentation, *even* if it mostly describes setuptools/distribute/pip. > >When people want to package a library, they *will* look for docs in Python, >and at the moment they only find the distutils reference. While the latter >is necessary, a more howto-like standard document is much needed. > >Georg I have the start of such a document, derived from my own recent experience pulling together all the information needed to release a new Python project. It's aimed at people new to packaging and possibly open source in general, so it's broader than just packaging. It includes things like basic file layout, unit testing, Pylint, getting started with Sphinx, a sample (distutils) setup.py, and how to register with PyPI. I try to pull together all the info a novice would need to have a reasonable shot at sharing their code. It's by no means all-encompassing (e.g. no mention of native extensions), but perhaps someone would be willing to write a complimentary advanced packaging howto. You can find it at: http://infinitemonkeycorps.net/docs/pph/ Let me know what you think. -John ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Thu, 12 Nov 2009 10:54:41 +0100, Tarek Ziadé wrote:
[clip]
>> get_install_paths('FOO')
>>>
>>> And that's the API we want to add in sysconfig, roughly.
>>
>> That does not solve the problem about getting FOO in the first place
>> when calling get_install_path.
>
> Why that ? where "FOO" comes from ? if it's an option you provide at
> build time like you said, earlier, you just pass it to the API to get
> the paths.
>
> FOO is not coming from nowhere...
If this is not painfully clear already, the user passes FOO to the command
python setup.py install --prefix=FOO
Now, clearly the distutils install subcommand knows what FOO is.
But does build_ext? Does sysconfig?
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Thu, Nov 12, 2009 at 5:30 AM, David Cournapeau
wrote:
> Greg Ewing wrote:
>> As far as I can tell David seems to be saying that
>> instantiating a class for every file in the system
>> would be too much overhead.
>>
>> I'm not convinced about that -- make builds a dependency
>> graph with a node for every target before it goes
>> to work, and I think scons does something similar
>
> Yes, and scons has scalability problems because of this (both from CPU
> and memory POV).
But you were saying in the discussion that Disutils has a *design*
problem because it uses OOP...
Anyways, let's not drop the other part of this thread:
> get_install_paths('FOO')
>>
>> And that's the API we want to add in sysconfig, roughly.
>
> That does not solve the problem about getting FOO in the first place
> when calling get_install_path.
Why that ? where "FOO" comes from ? if it's an option you provide at
build time like you said,
earlier, you just pass it to the API to get the paths.
FOO is not coming from nowhere...
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Greg Ewing wrote: > As far as I can tell David seems to be saying that > instantiating a class for every file in the system > would be too much overhead. > > I'm not convinced about that -- make builds a dependency > graph with a node for every target before it goes > to work, and I think scons does something similar Yes, and scons has scalability problems because of this (both from CPU and memory POV). Waf has also an object per node, but it is aggressively optimized. If you are interested in the details, I can point you to the corresponding discussions in both scons and waf, where the main designers were involved. > But that's not to say that a class couldn't be devised > that allowed the required flexibility without degenerating > into just a string with textual substitutions. I am not saying that's impossible, but both waf an scons use string susbtitution to do it (they do it differently though). So you would have to find a good reason not to do it IMHO. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Tarek Ziadé wrote: On Thu, Nov 12, 2009 at 12:03 AM, David Cournapeau wrote: You cannot obtain this with classes and objects (especially when you start talking about performance: thousand of source files for one extension is not a crazy usercase). Sorry, I am getting confused here. This is getting all mixed up. What OOP has to do with performance in the first place ? As far as I can tell David seems to be saying that instantiating a class for every file in the system would be too much overhead. I'm not convinced about that -- make builds a dependency graph with a node for every target before it goes to work, and I think scons does something similar (although I might be wrong, I haven't looked at scons very closely yet). >> $CC $CFLAGS $CPPDEFINES $CPPPATH -c $SOURCE $TARGET >> >> You need to be able to control those variable content in a very fine >> grained manner: prepending and appending may lead to different >> compiler behavior It's true that the current distutils compiler classes are lacking in flexibility here -- there's an option for "extra flags", for example, but it sticks them all at the end of the command. Sometimes that's wrong -- I've had trouble trying to use it for MacOSX -F arguments, for example. But that's not to say that a class couldn't be devised that allowed the required flexibility without degenerating into just a string with textual substitutions. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Robert Kern wrote: With all respect to Greg Ward and the rest of the original distutils authors, it was a fantastic improvement over the state of affairs ten years ago, but we've learned a lot about application frameworks and about building software since then. I think we knew more about it *before* then as well. Make has been around for a lot longer, but distutils ignores everything we've learned from it. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, Nov 11, 2009 at 11:22 PM, Robert Kern wrote: [..] > To get to one real specific problem, let's consider build_src. build_src is > a new subcommand in numpy.distutils that builds C extension sources from > other files. We use this to hook in f2py's wrapper generator and other more > ad hoc forms of generating wrappers. When build_ext uses --inplace, we need > build_src to use --inplace, too, because it will often output some "final" > products in addition to intermediate wrapper sources. In order to integrate > this with setuptools' develop command (which invokes build_ext --inplace but > not build_src --inplace because setuptools knows nothing about > numpy.distutils), we need to create a subclass of setuptool's develop > command that will reinitialize build_src with the appropriate option. Then > we need to conditionally place the develop command into the set of command > classes so as not to introduce a setuptools dependency on those people who > don't want to use it. > > This is nuts. This clearly indicates that we should be able to extend build_ext behaviour without subclassing it. And having the ability to drive a specific compiling from within an Extension subclass can solve this issue. > numpy.distutils shouldn't have to know anything about > setuptools to accomplish this if the framework were properly designed. And > this doesn't even get into the fact that many of the numpy.distutils command > classes that are shared with setuptools are conditional subclasses and > probably still buggily cobbled together. Same goes with "install", and I've proposed in the past the ability to run arbitrary commands as pre/post hooks for it. So we can "configure" this command instead of replacing it. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Thu, Nov 12, 2009 at 12:03 AM, David Cournapeau wrote:
> On Thu, Nov 12, 2009 at 5:42 AM, Tarek Ziadé wrote:
>
>>
>> I am not sure he wants FOO, I think he wants all installation paths,
>> that gets built
>> by the install command with the provided "FOO" root prefix.
>>
>> that could be in pseudo code:
>>
> get_install_paths('FOO')
>>
>> And that's the API we want to add in sysconfig, roughly.
>
> That does not solve the problem about getting FOO in the first place
> when calling get_install_path.
Why that ? where "FOO" comes from ? if it's an option you provide at
build time like you said,
earlier, you just pass it to the API to get the paths.
FOO is not coming from nowhere...
[..]
>
>> Do you mean, an Extension that would require several compilers ?
>>
>> I was thinking of a one-to-one relation between an Extension and a
>> compiler type,
>> even if there are are multiple source files (in different languages)
>> for this extension.
>
> This cannot work - you need different tools at different stages. I
> don't think any imperative method can work here. Note also that
> extension alone is not enough to trigger the right tool: it should be
> overridable on a per extension-basis (actually, it should be
> overridable on a per source basis if needed).
>
> I think compiler class and the likes should simply be removed of the
> picture here. You need tasks to transform a source into a target, and
> tasks would use compiler configuration. There should not be any
> objects/classes for compilers, it it not flexible enough.
> Although
> the details differ in significant ways, both waf and scons use strings
> for command lines (waf "compile them" for performance reason),
> consisting of different parts which can be altered at will. In scons,
> the task to compile C code is something like
>
> $CC $CFLAGS $CPPDEFINES $CPPPATH -c $SOURCE $TARGET
>
> You need to be able to control those variable content in a very fine
> grained manner: prepending and appending may lead to different
> compiler behavior, for example. This is especially important when
> linking, where the wrong order may be the different between a working
> extension and a crashing extension.
>
> You cannot obtain this with classes and objects (especially when you
> start talking about performance: thousand of source files for one
> extension is not a crazy usercase).
Sorry, I am getting confused here. This is getting all mixed up.
What OOP has to do with performance in the first place ?
OOP is useful to describe patterns and resuability, and that's what
Distutils does.
OOP is not a bottleneck here for speed. (everything is an object in
Python anyways)
Now we are saying that the compiler pattern in Distutils doesn't fit
some requirement,
fine, let's see what other design we can have.
But that will be based on OOP with classes, and objects, and there
ought to be an object that
orchestrates the building of one given extension at the end.
That might be the existing Extension class. This class is instanciated
with a name for the
extension, and a list of files.
Here's a proposal to restrict the scope: let's drop the concept of
"compiler class" and let's work only with
the Extension class.
This class is used by a distutils command to build a given extension.
Let's say it is standalone, and it has a unique "build()"
method. Could we take it from here and try to prototype this Extension class ?
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Hi Greg, On Thu, 12 Nov 2009 10:02:18 +1300, Greg Ewing wrote: > What we need right now, I think, is some discussion > about a new API, unconstrained by any considerations of > backwards compatibility A new API isn't so hard, but like anything it takes time and effort to do. Already this year, *a lot* of discussion went into such a thing. For example, doing metadata based installs as an option to traditional installs. Backwards compatibility is going to be an issue that *does* need to be addressed but even that I don't believe is so hard. There's two halves to "backwards" compatability. At the build side and at the install side. It's confusing I know. Where distutils/setup is confusing is at the build side in the way that all the files have to be specified for being picked up. On the install side, setup.py is much simpler and hardly* does anything at all except copy files in from the sources.txt. So compatability there is easier. It's my opinion that a boilerplate setup.py could completely replace 9/10 user written setup.py files. > When I say it's fundamentally broken, I'm really talking > about the API. My idea of what an API for a build system > should be like is more like make or scons, which slice > the functionality up in a completely orthogonal direction > to the way distutils does. yes. Broken on the build side. > Maybe it would be possible to plug such a system in under > the existing build_ext class. For even that to happen, there needs to be some work done. As in fleshing out some code and examples and documentation. Here's a configuration file for an alternate build system that I'm working on at: http://bitbucket.org/djlyon/pypi-package-builder Any help thrown at getting this working would be welcomed. Regards David setup.cfg [setup] name = artistflair version = 1.2 description = artistflair is a package written in spare time to show colour variations. [dependencies] packages = pyopengl objectrelationalmapper >= 1.6 [dependencies python<=2.4] packages = lxml >= 2.5 [dependencies windows] packages = win32com [dependencies linux] packages = pyodbc [Application] scripts = artisticflairgui.py [configuration] files = artisticflair.conf [datafiles] files = artisticdb.db directory = db [Documentation] directory = docs url = index.html [preinstall] script = checksystem.py [postinstall linux] script = builddocumentation.py [uninstall] script = uninstallcleanup.py - ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Thu, Nov 12, 2009 at 5:42 AM, Tarek Ziadé wrote:
>
> I am not sure he wants FOO, I think he wants all installation paths,
> that gets built
> by the install command with the provided "FOO" root prefix.
>
> that could be in pseudo code:
>
get_install_paths('FOO')
>
> And that's the API we want to add in sysconfig, roughly.
That does not solve the problem about getting FOO in the first place
when calling get_install_path.
Having a function which gives you all the build directories as well as
the install paths would already be great, though. Right now, for
numscons, I needed to reimplement all the logic, with the different
paths in different modes (inplace vs standard, Release vs Debug on
windows, etc...). Some of them are even python version dependent IIRC.
> Do you mean, an Extension that would require several compilers ?
>
> I was thinking of a one-to-one relation between an Extension and a
> compiler type,
> even if there are are multiple source files (in different languages)
> for this extension.
This cannot work - you need different tools at different stages. I
don't think any imperative method can work here. Note also that
extension alone is not enough to trigger the right tool: it should be
overridable on a per extension-basis (actually, it should be
overridable on a per source basis if needed).
I think compiler class and the likes should simply be removed of the
picture here. You need tasks to transform a source into a target, and
tasks would use compiler configuration. There should not be any
objects/classes for compilers, it it not flexible enough. Although
the details differ in significant ways, both waf and scons use strings
for command lines (waf "compile them" for performance reason),
consisting of different parts which can be altered at will. In scons,
the task to compile C code is something like
$CC $CFLAGS $CPPDEFINES $CPPPATH -c $SOURCE $TARGET
You need to be able to control those variable content in a very fine
grained manner: prepending and appending may lead to different
compiler behavior, for example. This is especially important when
linking, where the wrong order may be the different between a working
extension and a crashing extension.
You cannot obtain this with classes and objects (especially when you
start talking about performance: thousand of source files for one
extension is not a crazy usercase).
David
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On 2009-11-11 15:59 PM, Tarek Ziadé wrote: On Wed, Nov 11, 2009 at 10:36 PM, Robert Kern wrote: [..] It does feel something like that. The build system is just one of the problems with distutils' internals, in my experience. You can think of the rest of distutils as a little application framework for command line utilities. I think this framework simply fails to provide in very fundamental ways, like the "extend commands by subclassing" design pattern. That choice makes it fundamentally difficult to combine extensions together. I really don't see a way to evolve away from that (and believe me, over the last decade, I've tried). You just need to redesign the internals if you want to get away from that. You can't get from any point A to any point B by evolving in small steps that are functional (not to mention backwards compatible!) all the way through. I am very surprised about this statement. What did you tried for the paste decade and failed to do ? I hear some complaints since a week, but beside's David examples I didn't read any other precise use cases. We're looking through the build_ext use case, and we are making some improvement on the other thread. So why not doing this in other issues ? Let's discuss your use case. And if it means adding new options to run arbitrary commands like post/pre hooks to a given command, to avoid subclassing an existing command, let's do it. http://svn.scipy.org/svn/numpy/trunk/numpy/distutils/command/ All of it. Now consider that here we are also trying to play nicely with the setuptools extensions, and Pyrex, and David is working on integrating Cython support. To get to one real specific problem, let's consider build_src. build_src is a new subcommand in numpy.distutils that builds C extension sources from other files. We use this to hook in f2py's wrapper generator and other more ad hoc forms of generating wrappers. When build_ext uses --inplace, we need build_src to use --inplace, too, because it will often output some "final" products in addition to intermediate wrapper sources. In order to integrate this with setuptools' develop command (which invokes build_ext --inplace but not build_src --inplace because setuptools knows nothing about numpy.distutils), we need to create a subclass of setuptool's develop command that will reinitialize build_src with the appropriate option. Then we need to conditionally place the develop command into the set of command classes so as not to introduce a setuptools dependency on those people who don't want to use it. This is nuts. numpy.distutils shouldn't have to know anything about setuptools to accomplish this if the framework were properly designed. And this doesn't even get into the fact that many of the numpy.distutils command classes that are shared with setuptools are conditional subclasses and probably still buggily cobbled together. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, Nov 11, 2009 at 10:36 PM, Robert Kern wrote: [..] > > It does feel something like that. The build system is just one of the > problems with distutils' internals, in my experience. You can think of the > rest of distutils as a little application framework for command line > utilities. I think this framework simply fails to provide in very > fundamental ways, like the "extend commands by subclassing" design pattern. > That choice makes it fundamentally difficult to combine extensions together. > I really don't see a way to evolve away from that (and believe me, over the > last decade, I've tried). You just need to redesign the internals if you > want to get away from that. You can't get from any point A to any point B by > evolving in small steps that are functional (not to mention backwards > compatible!) all the way through. I am very surprised about this statement. What did you tried for the paste decade and failed to do ? I hear some complaints since a week, but beside's David examples I didn't read any other precise use cases. We're looking through the build_ext use case, and we are making some improvement on the other thread. So why not doing this in other issues ? Let's discuss your use case. And if it means adding new options to run arbitrary commands like post/pre hooks to a given command, to avoid subclassing an existing command, let's do it. And let's drop the backward compat issues in these discussions, so we don't burn out in details. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On 2009-11-11 15:02 PM, Greg Ewing wrote: Tarek Ziadé wrote: If the answer to that solution is just: "Distutils sucks anyways..", it is not really helpfull imho.. I don't see the point to write Distutils from scratch, instead of making it evolve. If you can see a way to get from the current distutils code to something with a well-designed and well-documented API and a clean implementation, that would be fine by me. When I say it's fundamentally broken, I'm really talking about the API. My idea of what an API for a build system should be like is more like make or scons, which slice the functionality up in a completely orthogonal direction to the way distutils does. Maybe it would be possible to plug such a system in under the existing build_ext class. I don't know. In fact, David has. http://pypi.python.org/pypi/numscons/ > I think I > would like the same philosophy applied to other areas > of distutils, not just compiling extensions. Otherwise > it would feel like two incompatible systems bolted > together. It does feel something like that. The build system is just one of the problems with distutils' internals, in my experience. You can think of the rest of distutils as a little application framework for command line utilities. I think this framework simply fails to provide in very fundamental ways, like the "extend commands by subclassing" design pattern. That choice makes it fundamentally difficult to combine extensions together. I really don't see a way to evolve away from that (and believe me, over the last decade, I've tried). You just need to redesign the internals if you want to get away from that. You can't get from any point A to any point B by evolving in small steps that are functional (not to mention backwards compatible!) all the way through. With all respect to Greg Ward and the rest of the original distutils authors, it was a fantastic improvement over the state of affairs ten years ago, but we've learned a lot about application frameworks and about building software since then. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé wrote: If the answer to that solution is just: "Distutils sucks anyways..", it is not really helpfull imho.. I don't see the point to write Distutils from scratch, instead of making it evolve. If you can see a way to get from the current distutils code to something with a well-designed and well-documented API and a clean implementation, that would be fine by me. When I say it's fundamentally broken, I'm really talking about the API. My idea of what an API for a build system should be like is more like make or scons, which slice the functionality up in a completely orthogonal direction to the way distutils does. Maybe it would be possible to plug such a system in under the existing build_ext class. I don't know. I think I would like the same philosophy applied to other areas of distutils, not just compiling extensions. Otherwise it would feel like two incompatible systems bolted together. What we need right now, I think, is some discussion about a new API, unconstrained by any considerations of backwards compatibility or reuse of existing distutils code. Once we know where we're going, then we can think about the best way to get there. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 9:22 PM, Pauli Virtanen wrote:
> ke, 2009-11-11 kello 16:47 +0100, Tarek Ziadé kirjoitti:
> [clip]
>> >> If it's the later, I guess you will be able to use the
>> >> upcoming "sysconfig" module,
>> >> that gives you the install schemes, depending on
>> >> sys.prefix/sys.exec_prefix.
>> >
>> > Where is the sysconfig sources ? I don't see it in bitbucket.
>>
>> That's in python's svn, in a tarek_sysconfig branch. It's a revamp of
>> distutils/sysconfig.py with the schemes from distutils/command/install.py
>> (work in progress)
>
> What if the user passes a different install prefix via
>
> python setup.py install --prefix=FOO
>
> I believe David would like to know FOO here. Since sysconf is not a part
> of distutils, will it know what FOO is?
I am not sure he wants FOO, I think he wants all installation paths,
that gets built
by the install command with the provided "FOO" root prefix.
that could be in pseudo code:
>>> get_install_paths('FOO')
And that's the API we want to add in sysconfig, roughly.
[..]
>
> I think one question here is that how do the different compilers speak
> to each other?
>
> Consider the following chain needed to compile a Cython file linking to
> some Fortran and C files to a Python module:
>
> - cython foo.pyx -> foo.c
> - compile foo.c -> foo.o
> - compile bar.c -> bar.o
> - compile quux.f90 -> quux.o
> - link foo.o bar.o quux.o -> foo.so
>
> This is a completely possible use-case, so it would be good if distutils
> could handle it. Also, dependency handling would be nice here. Changing
> bar.c or foo.pyx should trigger relinking foo.so etc.
>
>> So, yes, being able to register an arbitrary compiler class, with
>> arbitrary options passed through the Extension
>> could make it simpler:
>>
>> setup(
>> ..
>> ext_modules=[Extension('Foo', files=['foo.d'], compiler='pyd')],
>> ..)
>
> Does this work easily out when there are multiple source files (in
> different languages) for a single extension module?
Do you mean, an Extension that would require several compilers ?
I was thinking of a one-to-one relation between an Extension and a
compiler type,
even if there are are multiple source files (in different languages)
for this extension.
Meaning that *one* compiler would have to handle of those files. Does that fit
the cython use case ?
>
> Also, the option of determining the necessary compiler and compiler
> options from file extensions does not always work. For example, Fortran
> 90 files come in two flavors, fixed-form and free-form, which often are
> not differentiated by the file extension. However, they may require
> different compiler flags to compile. (Hopefully, however, nobody uses
> the fixed-form for F90 any more...)
How do they do then ? Is file extensions, inline options in setup.py,
and the environ, are enough ?
regards
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
ke, 2009-11-11 kello 16:47 +0100, Tarek Ziadé kirjoitti:
[clip]
> >> If it's the later, I guess you will be able to use the
> >> upcoming "sysconfig" module,
> >> that gives you the install schemes, depending on
> >> sys.prefix/sys.exec_prefix.
> >
> > Where is the sysconfig sources ? I don't see it in bitbucket.
>
> That's in python's svn, in a tarek_sysconfig branch. It's a revamp of
> distutils/sysconfig.py with the schemes from distutils/command/install.py
> (work in progress)
What if the user passes a different install prefix via
python setup.py install --prefix=FOO
I believe David would like to know FOO here. Since sysconf is not a part
of distutils, will it know what FOO is?
[clip]
> > For me, one of the core idea of an improved distutils would be to make
> > this much easier. All compilers options form distutils would be in
> > simple data files with simple API, no objects, no class with countless
> > methods and complex protocol. Distutils v2 would have a default "dumb"
> > build tool, but you could use whatever tool instead if desired.
>
> The default compiler class exists for that, it's CCompiler, and is
> mostly a placeholder for options.
> But it's C oriented. In my mind, implementing a new compiler for
> Distutils means overriding it, and implementing, mainly:
>
> - preprocess()
> - compile()
> - create_static_lib()
> - link()
>
> Now that's quite complex, and we could probably have a single method
> (compile) that would do the required work of compiling an extension
> the way it wants to.
I think one question here is that how do the different compilers speak
to each other?
Consider the following chain needed to compile a Cython file linking to
some Fortran and C files to a Python module:
- cython foo.pyx -> foo.c
- compile foo.c -> foo.o
- compile bar.c -> bar.o
- compile quux.f90 -> quux.o
- link foo.o bar.o quux.o -> foo.so
This is a completely possible use-case, so it would be good if distutils
could handle it. Also, dependency handling would be nice here. Changing
bar.c or foo.pyx should trigger relinking foo.so etc.
> So, yes, being able to register an arbitrary compiler class, with
> arbitrary options passed through the Extension
> could make it simpler:
>
> setup(
> ..
> ext_modules=[Extension('Foo', files=['foo.d'], compiler='pyd')],
> ..)
Does this work easily out when there are multiple source files (in
different languages) for a single extension module?
Also, the option of determining the necessary compiler and compiler
options from file extensions does not always work. For example, Fortran
90 files come in two flavors, fixed-form and free-form, which often are
not differentiated by the file extension. However, they may require
different compiler flags to compile. (Hopefully, however, nobody uses
the fixed-form for F90 any more...)
Cheers,
--
Pauli Virtanen
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
In line with this discussion, I found a document that details the aspects of CPAN that can be used for writing packaging systems in other languages. The author says: over the years people from at least Python, Ruby, and Java communities have approached me or other core CPAN people to ask basically "How did we do it? http://www.cpan.org/misc/ZCAN.html -srid On Fri, 06 Nov 2009 09:53:44 -0800, Guido van Rossum wrote: I just found this comment on my blog. People have told me this in person too, so I believe it is real pain (even if the solution may be elusive and the suggested solutions may not work). But I don't know how to improve the world. Is the work on distutils-sig going to be enough? Or do we need some other kind of work in addition? Do we need more than PyPI? --Guido -- Forwarded message -- From: dalloliogm Date: Fri, Nov 6, 2009 at 8:01 AM Subject: [Neopythonic] New comment on Python in the Scientific World. To: gvanros...@gmail.com dalloliogm has left a new comment on your post "Python in the Scientific World": Python is suffering a lot in the scientific word, because it has not a CPAN-like repository. PyPI is fine, but it is still far from the level of CPAN, CRAN, Bioconductor, etc.. Scientists who use programming usually have a lot of different interests and approaches, therefore it is really difficult to write a package that can be useful to everyone. Other programming language like Perl and R have repository-like structure which enable people to download packages easily, and to upload new ones and organize them withouth having to worry about having to integrate them into existing packages. This is what is happening to biopython now: it is a monolitic package that it is supposed to work for any bioinformatic problem; but this is so general that to accomplish that you would need to add a lot of dependencies, to numpy, networkx, suds, any kind of library. However, since easy_install is not as ready yet as the counterparts in other languages, if the biopython developers add too many dependencies, nobody will be able to install it properly, and nobody will use it. Posted by dalloliogm to Neopythonic at November 6, 2009 8:01 AM ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 3:47 PM, David Cournapeau wrote:
> On Wed, Nov 11, 2009 at 11:13 PM, Tarek Ziadé wrote:
>
>> Or it is just that you want to get the "--prefix" value finalized and
>> computed by the install
>> command.
>
> Yes.
Ok. What is obvious to me now is that the "install" command does to much.
Knowing the paths is something useful for any command. So "sysconfig"
will help I think.
>
>> If it's the later, I guess you will be able to use the
>> upcoming "sysconfig" module,
>> that gives you the install schemes, depending on sys.prefix/sys.exec_prefix.
>
> Where is the sysconfig sources ? I don't see it in bitbucket.
that's in python's svn, in a tarek_sysconfig branch. It's a revamp of
distutils/sysconfig.py
with the schemes from distutils/command/install.py
(work in progress)
[..]
>> I don't know for the first part. I have to try it out. Can you provide
>> me such an extension ?
>
> Not for make, but I can try to port numpy.distutils.command.scons to
> distutils (or distribute). The current code is damn ugly (I did most
> of it when I started digging into distutils), but you can get an idea
> here:
>
> http://github.com/cournape/numpy/blob/master/numpy/distutils/command/scons.py
>
> It calls scons, and you can thus build any C extension using scons.
> Now, both distutils and scons makes this difficult (in particular,
> there is no way to call scons from distutils, you need to launch scons
> executable).
I see. I'll take a look asap. Are you coming to Pycon btw ?
>
> For me, one of the core idea of an improved distutils would be to make
> this much easier. All compilers options form distutils would be in
> simple data files with simple API, no objects, no class with countless
> methods and complex protocol. Distutils v2 would have a default "dumb"
> build tool, but you could use whatever tool instead if desired.
The default compiler class exists for that, it's CCompiler, and is
mostly a placeholder for options.
But it's C oriented. In my mind, implementing a new compiler for
Distutils means overriding it, and implementing, mainly:
- preprocess()
- compile()
- create_static_lib()
- link()
Now that's quite complex, and we could probably have a single method
(compile) that would
do the required work of compiling an extension the way it wants to.
So, yes, being able to register an arbitrary compiler class, with
arbitrary options passed through the Extension
could make it simpler:
setup(
..
ext_modules=[Extension('Foo', files=['foo.d'], compiler='pyd')],
..)
where "pyd" is the name of the compiler that knows how to compile D files;
This compiler would do whatever it wants, as long as it is done in a
.compile() method:
.compile(name, files, *args, **kw)
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
2009/11/11 sstein...@gmail.com : > I don't think it's redundant because a lot of "needs" in Python go unmet, > not due to people's inability or unwillingness, but due to a lack of > time/funding which amount to the same thing. People *want* a lot of things. But if they truly *need* it, they'll do it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 14:16 +0100, Tarek Ziadé wrote: > On Wed, Nov 11, 2009 at 1:31 PM, David Cournapeau wrote: > But, at the end, since an option is either global, either specific to > a command, I guess a simple API in the command class should be enough > to avoid this hack: >get_option(command_name, option_name) +1 -- .''`. Wolodja Wentland : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 11:13 PM, Tarek Ziadé wrote: > Or it is just that you want to get the "--prefix" value finalized and > computed by the install > command. Yes. > If it's the later, I guess you will be able to use the > upcoming "sysconfig" module, > that gives you the install schemes, depending on sys.prefix/sys.exec_prefix. Where is the sysconfig sources ? I don't see it in bitbucket. > Sorry, I can see it yet, it still fuzzy. Does that mean your binary > distribution will not be relocatable ? Relinking was just an example, but yes, the binary would not relocatable in that case (although you can actually build relocatable binaries through $ORIGIN, but we are not here to talk about advanced deployment issues of binaries). Just to be clear, I am not advocating distutils to do it or even implement it at all, just to make it possible. > >>> This is similar to getting the command, (instanciate it + finalize it >>> if it doesn't exists yet) >>> and return a finalized option. >> >> This does not solve the issue IMHO. Since we both seem to like >> thinking about use-cases, consider this use-case: you have a python >> package with a complex extension built with make (say you have a >> build_make command which calls a makefile). How do you do it ? How to >> communicate path informations, compiler options between make and >> distutils ? How to handle relinking (changing rpath at install time) ? > > I don't know for the first part. I have to try it out. Can you provide > me such an extension ? Not for make, but I can try to port numpy.distutils.command.scons to distutils (or distribute). The current code is damn ugly (I did most of it when I started digging into distutils), but you can get an idea here: http://github.com/cournape/numpy/blob/master/numpy/distutils/command/scons.py It calls scons, and you can thus build any C extension using scons. Now, both distutils and scons makes this difficult (in particular, there is no way to call scons from distutils, you need to launch scons executable). For me, one of the core idea of an improved distutils would be to make this much easier. All compilers options form distutils would be in simple data files with simple API, no objects, no class with countless methods and complex protocol. Distutils v2 would have a default "dumb" build tool, but you could use whatever tool instead if desired. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 2:48 PM, David Cournapeau wrote: >> If you want to have install options for your command, and if your >> command is about "installing", > > I did not say my command was about installing - it does not install > anything. To be complete, we do not use this in build_clib, but in > build_src , to generate pkg-config-like files (build_src is a > numpy.distutils-specific command to build source files, for example > automatically generated wrappers around fortran libraries, and is > typically run before any other build_ command) > But you call it with "install" in your example, meaning that is is called at install time, right ? Or it is just that you want to get the "--prefix" value finalized and computed by the install command. If it's the later, I guess you will be able to use the upcoming "sysconfig" module, that gives you the install schemes, depending on sys.prefix/sys.exec_prefix. And I will probably add a way to override those prefix, meaning that you will be able to get from your command all install paths depending on a root prefix. If not, it means that you are doing a post or pre install step during installation. e.g. like RPM's pre/post commits hooks. >> >> For build_clib, if you need to build something, it goes in the >> build_dir, or in place., and this is not impacted >> by the install command. > > That's exactly the problem. build + install description is too > simplistic for complex builds: you often need to know install options > at build time. You need to know some build options when building > configuration files, you need to pass configuration options to other > subsequent steps, etc... Sorry, I can see it yet, it still fuzzy. Does that mean your binary distribution will not be relocatable ? e.g. meaning that once you have done your build for a given --prefix, it won't be installable anywhere else ? In that case you would need to remove the --prefix option from install, and have it on your build command. Sounds like a pre/post install hook but I am not sure. >> This is similar to getting the command, (instanciate it + finalize it >> if it doesn't exists yet) >> and return a finalized option. > > This does not solve the issue IMHO. Since we both seem to like > thinking about use-cases, consider this use-case: you have a python > package with a complex extension built with make (say you have a > build_make command which calls a makefile). How do you do it ? How to > communicate path informations, compiler options between make and > distutils ? How to handle relinking (changing rpath at install time) ? I don't know for the first part. I have to try it out. Can you provide me such an extension ? For the relinking at installation time problem, it is obvious that something can be done. we could have a pre/post install option where an arbitrary command could be launch, as an install subcommand. pre: works on the build_dir built by install or a previous build. post: works on the installed file list [..] >> If you are willing to spend some time in that, I am the guy who can >> commit your patches in Python. > > I will try to prepare a couple of patches against the hg repo this WE, yeah, thanks ! \o/ Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 10:16 PM, Tarek Ziadé wrote: > > Ouch, That's not to be done. Something is wrong with your build_clib > design here. > You are roughly calling "install" as a sub command. Yes, it is "wrong" from distutils POV, but there is no other solution to get the prefix. The problem is that prefix is available in install, not that I want to know the prefix when building. > > If you want to have install options for your command, and if your > command is about "installing", I did not say my command was about installing - it does not install anything. To be complete, we do not use this in build_clib, but in build_src , to generate pkg-config-like files (build_src is a numpy.distutils-specific command to build source files, for example automatically generated wrappers around fortran libraries, and is typically run before any other build_ command) > > For build_clib, if you need to build something, it goes in the > build_dir, or in place., and this is not impacted > by the install command. That's exactly the problem. build + install description is too simplistic for complex builds: you often need to know install options at build time. You need to know some build options when building configuration files, you need to pass configuration options to other subsequent steps, etc... > > This is similar to getting the command, (instanciate it + finalize it > if it doesn't exists yet) > and return a finalized option. This does not solve the issue IMHO. Since we both seem to like thinking about use-cases, consider this use-case: you have a python package with a complex extension built with make (say you have a build_make command which calls a makefile). How do you do it ? How to communicate path informations, compiler options between make and distutils ? How to handle relinking (changing rpath at install time) ? >> >> I will try to document how scons does it. I think the basic idea could be >> reused in distutils. > > But, you didn't answer: if we add the ability to define a different > compiler for each Extension, > will it solve your use case ? I did not answer because my answer will take time to accurately summarize tools problems in build tools. We have discussed quite a bit about those issues in scons as well and I want to be sure that my answer takes this into account as well. > > If you are willing to spend some time in that, I am the guy who can > commit your patches in Python. I will try to prepare a couple of patches against the hg repo this WE, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Nov 11, 2009, at 7:38 AM, Lennart Regebro wrote: 2009/11/11 sstein...@gmail.com : On Nov 11, 2009, at 4:39 AM, Lennart Regebro wrote: If nobody is willing to write the code, the code is not needed. I think it would be more accurate to say that nobody is deciding that the "need" is sufficient to invest the resources to fill it. More accurate, but longer and redundant. :) I don't think it's redundant because a lot of "needs" in Python go unmet, not due to people's inability or unwillingness, but due to a lack of time/funding which amount to the same thing. S ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
On Wed, Nov 11, 2009 at 1:31 PM, David Cournapeau wrote:
[..]
> AFAIK, there is only one way to get the information: since the --prefix is
> only
> known once install.finalize_options() has run, you need to call the method
> from
> build_clib.
>
> Naive code could be something like
>
> class MyBuildClib(build_clib):
> def run(self):
> install_cmd = self.distribution.get_command_obj("install")
> if not install_cmd.finalized == 1:
> install_cmd.finalize_options()
>
> if self.inplace == 1:
> top = "."
> else:
> top = install_cmd.install_libbase
> # top is the top directory where libraries should be installed
> build_clib.run(self)
>
> It is already quite ugly, but it looks like it could work. It actually does in
> the simple case, like python setup.py install. But if you call python setup.py
> build_clib install --prefix=something, it does not. The error message is a
> typical example of distutils style: "error: must supply either prefix/exec-
> prefix/home or install-base/install-platbase -- not both". The only way I
> managed to make this work is to replace:
Ouch, That's not to be done. Something is wrong with your build_clib
design here.
You are roughly calling "install" as a sub command.
If you want to have install options for your command, and if your
command is about "installing",
it means that your command has to be a subcommand of "install". Those
get called once the options
passed to install have been finalized.
IOW, build_clib is not a subcommand of install, so you have troubles.
subcommands are: install_lib, install_headers, install_scripts,
install_data, install_egg_info
(and I agree that it's not simple to extend this list)
But, why in the first place, do you need the install --prefix in a
build command ?
For build_clib, if you need to build something, it goes in the
build_dir, or in place., and this is not impacted
by the install command.
>
> If you have a configure / build / install like the vast majority of build
> systems out there, this problem disappears. I don't see how the problem can be
> fixed without touching how commands work.
I fail to understand your demonstration. Commands that are in charge
of the *build*
have nothing to do with commands that are in charge of *installing*
file in various places
in the target system. So I fail to understand why build_clib interacts
with install,
and why it has to impact it howsoever (or vice-versa)
Now, if we take the generic use case (even if I don't think it should
be used in your case):
"a simple way to share options amongst commands"
As a matter of fact, there's a hackish way to perform this, by using
the distribution instance
as a placeholder for these "common" options, so several command can
read/write it.
(as opposed to local options and global options) without having to get
the command that manage the option.
But, at the end, since an option is either global, either specific to
a command, I guess a simple API in the command class should be enough
to avoid this hack:
get_option(command_name, option_name)
This is similar to getting the command, (instanciate it + finalize it
if it doesn't exists yet)
and return a finalized option.
>
> Moreover, that's a typical example where the only way to be robust is to check
> that every attribute you are using actually exist before. At that point, you
> really want to run back to m4, perl, autogenerated makefiles and shell
> programming :)
>
>> I am still waiting for you comment on the solution to remove the
>> compile code from build_ext
>> and have it at the Extension level, with an option to add new
>> compilers in a more easier way.
>
> I will try to document how scons does it. I think the basic idea could be
> reused in distutils.
But, you didn't answer: if we add the ability to define a different
compiler for each Extension,
will it solve your use case ?
>> If there's someone from the Numpy project that can help in isolating patches
>> againts Distutils trunk in our issue tracker, I'd be more than happy
>> to reduce the gap between
>> the two projects.
>
> If that was not clear, I am that guy. I have been the main
> maintainer of numpy.distutils and of numpy/scipy build infrastructure for some
> time now,
If you are willing to spend some time in that, I am the guy who can
commit your patches in Python.
Regards
Tarek
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
2009/11/11 sstein...@gmail.com : > On Nov 11, 2009, at 4:39 AM, Lennart Regebro wrote: > >> If nobody is willing to write the code, the code is not needed. > > I think it would be more accurate to say that nobody is deciding that the > "need" is sufficient to invest the resources to fill it. More accurate, but longer and redundant. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Nov 11, 2009, at 4:39 AM, Lennart Regebro wrote: If nobody is willing to write the code, the code is not needed. I think it would be more accurate to say that nobody is deciding that the "need" is sufficient to invest the resources to fill it. There are lots of things that many people would be willing and able to write, that would fill a great "need", and that many people would benefit from, but that aren't getting written because there's no one willing to invest the time or money. In come cases people are both willing and able, but unable to invest the time due to the pressing need to make a living. I'd love to work on open source all day, but my wife and kids get cranky if they don't eat for a couple of days. People don't always agree on what is the "rational" need to fill, and sometimes what we, as programmers "need", and know would make us more productive, is not what the people controlling the pursestrings are willing to finance, even if it would ultimately benefit them financially. S ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Tarek Ziadé gmail.com> writes:
>
> Also, notice that we are in the process of adding a new python module
> in the stdlib, called
> "sysconfig", that will contains all installation paths scheme for all
> supported platforms.
I don't think we are talking about the same thing. If the pb was just missing
API, we could have just added it to numpy.distutils. We override pretty much
every command of distutils and setuptools anyway.
I needed to obtain the install prefix, be it the default one, or the one set
through --prefix, or the current directory for in place build/develop install.
I need to know it when building C libraries, in build_clib.
AFAIK, there is only one way to get the information: since the --prefix is only
known once install.finalize_options() has run, you need to call the method from
build_clib.
Naive code could be something like
class MyBuildClib(build_clib):
def run(self):
install_cmd = self.distribution.get_command_obj("install")
if not install_cmd.finalized == 1:
install_cmd.finalize_options()
if self.inplace == 1:
top = "."
else:
top = install_cmd.install_libbase
# top is the top directory where libraries should be installed
build_clib.run(self)
It is already quite ugly, but it looks like it could work. It actually does in
the simple case, like python setup.py install. But if you call python setup.py
build_clib install --prefix=something, it does not. The error message is a
typical example of distutils style: "error: must supply either prefix/exec-
prefix/home or install-base/install-platbase -- not both". The only way I
managed to make this work is to replace:
install_cmd = self.distribution.get_command_obj("install"))
by
install_cmd = copy.copy(self.distribution.get_command_obj("install"))
That's not an API problem, or a documentation problem. That's a fundamental
issue because --prefix is only known from install, and when you start running
commands in an order different than expected by distutils, you get weird errors
(the above error is actually almost sensical if you really know distutils
code). I passed over the fact that in some conditions that elude me ATM and are
platform specific, install_libbase does not exist and raises an attribute
error.
If you have a configure / build / install like the vast majority of build
systems out there, this problem disappears. I don't see how the problem can be
fixed without touching how commands work.
Moreover, that's a typical example where the only way to be robust is to check
that every attribute you are using actually exist before. At that point, you
really want to run back to m4, perl, autogenerated makefiles and shell
programming :)
> I am still waiting for you comment on the solution to remove the
> compile code from build_ext
> and have it at the Extension level, with an option to add new
> compilers in a more easier way.
I will try to document how scons does it. I think the basic idea could be
reused in distutils.
> If there's someone from the Numpy project that can help in isolating patches
> againts Distutils trunk in our issue tracker, I'd be more than happy
> to reduce the gap between
> the two projects.
If that was not clear, I am that guy. I have been the main
maintainer of numpy.distutils and of numpy/scipy build infrastructure for some
time now,
David
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, 11 Nov 2009 19:14:42 +1300, Greg Ewing wrote: > Also, it seems to me that in this case, the basic > architecture of distutils is already so full of > mistakes that there just isn't an incremental way > of getting to a better place, especially given the > requirement of not breaking any existing setup.py > scripts together with the fact that the API of distutils > effectively consists of its *entire implementation*. > > So while complete rewrites are best avoided *if possible*, > I don't think we have a choice in this case. Fwiw, I can't see any way to make that work. Even if I said that I didn't like distutils, which btw isn't true because my knowledge of it is so limited, it's just not practical to rewrite such a complex tool within any reasonable timeframe. Anyway, moving towards CPAN isn't about rewriting all of distutils, only a few key parts. Or restructuring. The parts that shouldn't be rewritten (because the people that wrote it were too clever - and the degree of difficulty to high) is any C interface. That's an important part of distutils that is likely to end up worse rather than better. If there are candidates for 'evolution', it has to be in the areas such as package creation (collecting all the files and putting them in a .tar.gz or .egg or .zip or .exe) for upload to pypi. In CPAN, one wouldn't dream of there being so many possibilities. As I've said on python-dev, we should just revise and call a package an egg. In science, it wouldn't be tolerated to call a hydrogen atom using 6 different synonyms. But in python we do. Then we wonder why people get confused... There's a big difference between a rewrite and an evolution. Rewrite isn't viable, but an evolution is. >From science, we even know that it only takes two generations to become nearly immune from strong radiation as shown in chernoble. So when we talk about evolution lets use it in the modern sense, not in the darwinian. Evolution is better than revolution is better than nothing happening at all. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, Nov 11, 2009 at 10:46 AM, David Cournapeau wrote: [..] > Example: how to retrieve the install prefix in setup.py. You need a good > understanding of distutils to understand why it is so complicated, and > the example shows almost everything that's wrong in distutils design. > Many expectations are undocumented, like what method/class can be called > where and when. All this implicit behavior is part of the API, and > that's not documented anywhere that I know of. Ok, so I read this example as a lack of documentation and the lack of clear APIs to get the installation paths. Also, notice that we are in the process of adding a new python module in the stdlib, called "sysconfig", that will contains all installation paths scheme for all supported platforms. I have a branch in the svn going on for that. Knowing where to install things and what is up with a given platform is a wider problem that Distutils in fact, it concerns site.py as well, and having a sysconfig module that handles this will help. I also expect it to help the work people from Jython, PyPy, and IronPython will do in the future. > > Also, what happened in python 2.6.3/2.6.4 w.r.t. distribute has happened > quite often for numpy.distutils, and I consider it inherent to distutils > way of working. I fully agree that this particular example is demonstrating that build_ext is suffering from a lack of API documentation. I am still waiting for you comment on the solution to remove the compile code from build_ext and have it at the Extension level, with an option to add new compilers in a more easier way. > >> If the Numpy projects made some refactoring/improvement, why the >> project didn't try to push >> it back in Distutils itself ? > > They are not improvements or refactoring for most part. They are things > quite specific to our needs: fortran support, support for our internal > code generator, swig, f2py, etc... A few things could be useful to > distutils, like manifest support for mingw for 2.6 and later, as well as > basic autoconf-like tests (checking for functions, type, type size, > etc...). We would be happy to contribute patches to Distribute if this > is considered useful. If there's someone from the Numpy project that can help in isolating patches againts Distutils trunk in our issue tracker, I'd be more than happy to reduce the gap between the two projects. Notice that this may have been done in the past, I didn't manage to review all the distutils bugs yet (some are 5 years old) Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé wrote: > On Wed, Nov 11, 2009 at 7:05 AM, David Cournapeau > wrote: > [..] >> Exactly. The fact that we in numpy consider distutils backward >> compatibility not worth the cost, even though we are most likely the >> most tied up with distutils, is quite telling about the state of affairs >> IMHO. > > That doesn't prove Distutils can't evolve. No, but that's not the point I was trying to make: I meant that we consider the distutils API not to be a significant asset, and would be happy to throw away all numpy.distutils for a significantly better system. We now have two build systems in numpy (one based on scons): I think it takes me 5 to 10 times more effort on average to add a feature to the distutils-based compared to the scons system. There are some features I can not implement because I have not find a solution to it in distutils. There are arbitrary limitations like the inability to call commands directly from setup.py, retrieve global informations from setup.py, classes which behave differently on different platforms. Example: how to retrieve the install prefix in setup.py. You need a good understanding of distutils to understand why it is so complicated, and the example shows almost everything that's wrong in distutils design. Many expectations are undocumented, like what method/class can be called where and when. All this implicit behavior is part of the API, and that's not documented anywhere that I know of. Also, what happened in python 2.6.3/2.6.4 w.r.t. distribute has happened quite often for numpy.distutils, and I consider it inherent to distutils way of working. > If the Numpy projects made some refactoring/improvement, why the > project didn't try to push > it back in Distutils itself ? They are not improvements or refactoring for most part. They are things quite specific to our needs: fortran support, support for our internal code generator, swig, f2py, etc... A few things could be useful to distutils, like manifest support for mingw for 2.6 and later, as well as basic autoconf-like tests (checking for functions, type, type size, etc...). We would be happy to contribute patches to Distribute if this is considered useful. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
To the people who wants a rewrite, two things need to be asked: 1. Do you think the new PEPs in development should be followed? In that case, what is the benefit of rewriting, instead of fixing? 2. When are you done? :-) I'm not being rude, but this is open source. There is no "Someone" that can rewrite distutils from scratch, it must be done by those who thinks it should be done. Those who think distutils should be rewritten from scratch need to sit down and do it. If nobody is willing to write the code, the code is not needed. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, Nov 11, 2009 at 7:05 AM, David Cournapeau wrote: [..] > > Exactly. The fact that we in numpy consider distutils backward > compatibility not worth the cost, even though we are most likely the > most tied up with distutils, is quite telling about the state of affairs > IMHO. That doesn't prove Distutils can't evolve. That just shows that numpy worked on its side. If the Numpy projects made some refactoring/improvement, why the project didn't try to push it back in Distutils itself ? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, Nov 11, 2009 at 7:14 AM, Greg Ewing wrote: > David Cournapeau wrote: >> >> One of the >> main argument to avoid rewrite is that you will end up doing the same >> mistakes, and old code is more complicated because it has been tested. >> But here, we know what a good design is like as other languages have >> vastly superior solutions to this problem. > > Also, it seems to me that in this case, the basic > architecture of distutils is already so full of > mistakes that there just isn't an incremental way > of getting to a better place, especially given the > requirement of not breaking any existing setup.py > scripts together with the fact that the API of distutils > effectively consists of its *entire implementation*. > > So while complete rewrites are best avoided *if possible*, > I don't think we have a choice in this case. While "build_ext" is not handy, I don't buy the fact that Distutils is "full of mistakes". We have to work with use cases. David gave a use case: being able to compile cython or assembly files. I proposed a solution based on being able to define a compiler at the Extension level, rather that for the entire build_ext command. If the answer to that solution is just: "Distutils sucks anyways..", it is not really helpfull imho.. I don't see the point to write Distutils from scratch, instead of making it evolve. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Wed, Nov 11, 2009 at 3:08 AM, David Cournapeau wrote: [..] > > What is important here is how to add new tools, without touching or > impacting distutils in many places. In particular, what distutils > expects from a tool should be clearly defined (right now it is > implementation defined). Yes, and in that case, it means writing a new compiler class. [...] >> Distutils is not living only through setup.py. It has some public APIs >> imported and used by people. > > I am aware about the usage of distutils: I don't think it has public > API, though. It has functions that people use, but it is far from clear > what is public from what is private. Many things we do numpy.distutils > are clearly dependent on implementation details of distutils. > A public api doesn't have a "_" prefix. [..] > Otherwise, a new system would look nothing like distutils. One of the > main argument to avoid rewrite is that you will end up doing the same > mistakes, and old code is more complicated because it has been tested. > But here, we know what a good design is like as other languages have > vastly superior solutions to this problem. > > As far as compilation is concerned at least, the distutils knowledge is > vastly overblown. First, most of it comes from autoconf on unices. You > have the MSVC tools knowledge, but that can be easily reused (in > numscons, I added msvc support from the python 2.6 msvc9compiler, this > was not difficult). Most other tools are rather obsolete - and if you > break any API in distribute there, you will most likely lose them as > well anyway (I am thinking about OS/2, Metrowerk kind of tools). > > Again, I don't mean to say that working on distribute is a mistake, or > criticize what you do in any way. I just don't think it will solve any > significant issue for the scientific python community. But enough > "stop-energy": at this point, I will just shut up, and continue working > on my own idea - if they make sense, the scientific community is big > enough so that the solution could be used there only, at least for some > time. I asked you for your use cases so we could work on changing things, but it's evident at this point that you don't want to use Distutils or you don't think it can evolve. I don't think the scientific community is so different from any other Python community either, in what they need. And I don't think Distutils is a lost case, as you seem to think. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Greg Ewing wrote: > > Also, it seems to me that in this case, the basic > architecture of distutils is already so full of > mistakes that there just isn't an incremental way > of getting to a better place, especially given the > requirement of not breaking any existing setup.py > scripts together with the fact that the API of distutils > effectively consists of its *entire implementation*. Exactly. The fact that we in numpy consider distutils backward compatibility not worth the cost, even though we are most likely the most tied up with distutils, is quite telling about the state of affairs IMHO. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
David Cournapeau wrote: One of the main argument to avoid rewrite is that you will end up doing the same mistakes, and old code is more complicated because it has been tested. But here, we know what a good design is like as other languages have vastly superior solutions to this problem. Also, it seems to me that in this case, the basic architecture of distutils is already so full of mistakes that there just isn't an incremental way of getting to a better place, especially given the requirement of not breaking any existing setup.py scripts together with the fact that the API of distutils effectively consists of its *entire implementation*. So while complete rewrites are best avoided *if possible*, I don't think we have a choice in this case. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé wrote: > On Tue, Nov 10, 2009 at 2:31 AM, David Cournapeau > wrote: > [..] >> If you want to support a new source/target (say assembler), you need to >> deal at this stage (or use hacks to deal with the new file extension >> you want to deal with, and remove it later...). So you need to copy >> build_ext, you cannot extend it. >> >> If you look at msvccompiler vs msvc9compiler, both compile methods are >> almost the same and copied from each other. > > So here's a refactoring proposal : > > - build_ext doesn't handle a compiler instance anymore (therefore it > doesn't have a "compile_type" anymore) > > - instead, the Extension class handles this compile type > > - right now we have a dict in ccompiler (compiler_class) that has all > the compiler classes. > let's make it extensible through a registery using distutils.cfg > (like the commands) > > so: when build_ext is run, it compiles the Extensions by using this > registery of compilers, and the type of compiler contained in each > extension. It keeps instances in a pool to reuse them if needed/ What is important here is how to add new tools, without touching or impacting distutils in many places. In particular, what distutils expects from a tool should be clearly defined (right now it is implementation defined). Adding cython or assembly support are good use cases to refactor this I think. > You just don't replace a stdlib package with its v2 like that, there > is a deprecation process to have. > > Distutils is not living only through setup.py. It has some public APIs > imported and used by people. I am aware about the usage of distutils: I don't think it has public API, though. It has functions that people use, but it is far from clear what is public from what is private. Many things we do numpy.distutils are clearly dependent on implementation details of distutils. The problem with moving forward distutils is not deprecation, the problem is when you break/remove the old API. At this point, you will break people's code who rely on the old API. If the changes are big enough (as they should be to warrant breaking the old API in the first place), it will require significant effort from distutils API users. So people will see this as a trade-off: does it worth my time to spend time to use the new version of distutils, what will it bring to me ? Without significant new features, this will be difficult. > -1 > > I'm hearing this sometimes. That's what I thought too when I started > to work with Distutils > a year and a half ago. And I couldn't understand Guido when he was saying that > redoing it from scratch would be a mistake. Now I agree with him. IIRC, the main argument was that a new system would mean giving up existing setup.py, but that can be worked around. I don't know whether Guido's opinion has changed since, but he wondered last year wether dropping backward compatibility was an option (http://www.opensubscriber.com/message/distutils-sig@python.org/10291496.html). Otherwise, a new system would look nothing like distutils. One of the main argument to avoid rewrite is that you will end up doing the same mistakes, and old code is more complicated because it has been tested. But here, we know what a good design is like as other languages have vastly superior solutions to this problem. As far as compilation is concerned at least, the distutils knowledge is vastly overblown. First, most of it comes from autoconf on unices. You have the MSVC tools knowledge, but that can be easily reused (in numscons, I added msvc support from the python 2.6 msvc9compiler, this was not difficult). Most other tools are rather obsolete - and if you break any API in distribute there, you will most likely lose them as well anyway (I am thinking about OS/2, Metrowerk kind of tools). Again, I don't mean to say that working on distribute is a mistake, or criticize what you do in any way. I just don't think it will solve any significant issue for the scientific python community. But enough "stop-energy": at this point, I will just shut up, and continue working on my own idea - if they make sense, the scientific community is big enough so that the solution could be used there only, at least for some time. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Tue, Nov 10, 2009 at 2:31 AM, David Cournapeau wrote: [..] > > If you want to support a new source/target (say assembler), you need to > deal at this stage (or use hacks to deal with the new file extension > you want to deal with, and remove it later...). So you need to copy > build_ext, you cannot extend it. > > If you look at msvccompiler vs msvc9compiler, both compile methods are > almost the same and copied from each other. So here's a refactoring proposal : - build_ext doesn't handle a compiler instance anymore (therefore it doesn't have a "compile_type" anymore) - instead, the Extension class handles this compile type - right now we have a dict in ccompiler (compiler_class) that has all the compiler classes. let's make it extensible through a registery using distutils.cfg (like the commands) so: when build_ext is run, it compiles the Extensions by using this registery of compilers, and the type of compiler contained in each extension. It keeps instances in a pool to reuse them if needed/ [..] > > I have given very explicit examples in this discussion - I have written > them on the wiki last time it was discussed as requested: > > http://wiki.python.org/moin/Distutils/PluginSystem > > I don't think it is accurate to summarize my critic as a vague "do > imperative operations". Sorry about that, I lost track of this. > >> >> Last, I am not sure why you want only backward-compatible changes in >> distutils. >> >> There's no plan to keep backward-compatibility if breaking it makes >> DIstutils better. We will have pending deprecation warnings, that's >> all. > > What is the intended policy: deprecate something in v1 and break it in > v1+1 ? I am not sure this works well when you need to correct deep > design issues which impact a lot of code (and this will be required). > For example, I would prefer not changing numpy.distutils several times > just to make a few API cleaner. You just don't replace a stdlib package with its v2 like that, there is a deprecation process to have. Distutils is not living only through setup.py. It has some public APIs imported and used by people. While some deep changes are required, if setting up a deprecation is possible, I will do it. > > I guess I don't see the point of breaking things while keeping the > current distutils codebase. I would rather do the exact contrary: throw > away the codebase, but keep current setup.py working through a > conversion script. Things like sandboxing pypi and other things become > much easier that way. -1 I'm hearing this sometimes. That's what I thought too when I started to work with Distutils a year and a half ago. And I couldn't understand Guido when he was saying that redoing it from scratch would be a mistake. Now I agree with him. There's a *lot* of knowledge in Distutils. Throwing it away means entering a new cycle that will last for several years. Small refactoring can be done today, as long as the code is tested. While the code is not yet as modern as other modules in the stdlib, I invite you to compare Python 2.4's distutils with the current trunk, and realize the amount of work I've been doing in the past year. My goal was to have a good coverage before starting any big refactoring, and to fix the old pending bugs 2 years ago, the code was barely tested (the coverage was 18%) and now it's well tested (80%+ depending on the platform). It's now almost fully PEP 8, and I am now more confident when I start any refactoring. I understand the pain with build_ext. let's work it out. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé wrote:
>
> What is a registery of extension exactly ? Distutils let you register your own
> commands, you can use through the CLI.
>
> Can you provide more details ?
Sure. Right now, if you want to deal with a new source or a new target,
you need to create a new command or override one. For example, cython
has distutils extension which subclass build_ext, we in numpy do the
same. This causes several issues:
- cython build_ext subclass distutils command, we do the same. How
to use both at the same time ? That's one typical example of the
subclassing issue Ian Bicking mentioned,
- in the case of compiled extensions, the basic structure is to
build all object files for each extension from the compiler class
(compile method). The compile method structure is:
for obj in objects:
if src == "bla":
do_bla()
elif src == "blabla":
do_blabla()
...
else:
raise some exception for unrecognied extension.
If you want to support a new source/target (say assembler), you need to
deal at this stage (or use hacks to deal with the new file extension
you want to deal with, and remove it later...). So you need to copy
build_ext, you cannot extend it.
If you look at msvccompiler vs msvc9compiler, both compile methods are
almost the same and copied from each other.
Now, if instead you have a dictionary {src_extension: callable}, you
would be able to add a new tool and extend existing code without
touching or copying anything. You could have syntactic sugar to have
something like:
@extension(".asm")
def assemble():
do_assembler_stuff()
All the assembler-related thing would be in a separate module, and you
would only need to register for the tool explicitly when you need it
(every tool could be the same, and current distutils tools would then be
registered automatically).
> I am in for making Distutils evolve, but I need very precise real
> world use cases
> not saying that Distutils shouldn't do imperative operations).
I have given very explicit examples in this discussion - I have written
them on the wiki last time it was discussed as requested:
http://wiki.python.org/moin/Distutils/PluginSystem
I don't think it is accurate to summarize my critic as a vague "do
imperative operations".
>
> Last, I am not sure why you want only backward-compatible changes in
> distutils.
>
> There's no plan to keep backward-compatibility if breaking it makes
> DIstutils better. We will have pending deprecation warnings, that's
> all.
What is the intended policy: deprecate something in v1 and break it in
v1+1 ? I am not sure this works well when you need to correct deep
design issues which impact a lot of code (and this will be required).
For example, I would prefer not changing numpy.distutils several times
just to make a few API cleaner.
I guess I don't see the point of breaking things while keeping the
current distutils codebase. I would rather do the exact contrary: throw
away the codebase, but keep current setup.py working through a
conversion script. Things like sandboxing pypi and other things become
much easier that way.
cheers,
David
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN
Guido van Rossum python.org> writes: > > I just found this comment on my blog. People have told me this in > person too, so I believe it is real pain (even if the solution may be > elusive and the suggested solutions may not work). But I don't know > how to improve the world. Is the work on distutils-sig going to be > enough? Or do we need some other kind of work in addition? Do we need > more than PyPI? > > --Guido I didn't see this mentioned in the other comments: Sometimes easy_install just gets stuck with an IncompleteRead error or something like that. (I may be butchering the error's name, sorry.) Then some part of the html of the page gets printed up to a seemingly arbitrary point, and the install fails. I'm not sure what causes that, but I've seen it happening at random for at least a couple of months. Sometimes it works, sometimes it fails with the incomplete read thing, and then if you try again it will fail at the same point of the html page. Try it a few hours later, it might work then. Not nice. (I've seen it happen on both Linux and Windows.) Ram. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On 11/9/09, David Cournapeau wrote: [...] > Some things are fixable in distutils: for example, to build things, you > should be able to get rid of the imperative operations, and have instead > of registry of extension -> action (ala scons/waf). What is a registery of extension exactly ? Distutils let you register your own commands, you can use through the CLI. Can you provide more details ? >> I'd really be shocked if a >> rewrite of distutils was necessary, or even necessary to simplify >> things. > > That's the opposite of my own experience. I think I have given several > reasonable examples of shortcomings of distutils: I would be glad to > hear that each of them has a simple solution which is > backward-compatible in distutils. I am in for making Distutils evolve, but I need very precise real world use cases not saying that Distutils shouldn't do imperative operations). Last, I am not sure why you want only backward-compatible changes in distutils. There's no plan to keep backward-compatibility if breaking it makes DIstutils better. We will have pending deprecation warnings, that's all. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
> I suggest that we check for valid metadata on the uploaded sdists at the > least. If you visit > http://pypm.activestate.com/ - most failed packages are due to the fact the > sdist > uploaded by the author misses certain files such as README.txt (that is read > by setup.py) > or setup.py/PKG-INFO itself. Unfortunately we can't run arbitrary code on PyPI. So if someone ships a broken setup.py, there's nothing much we can do unless we are able to run it in some kind of jail. Some work was started with Steve Steiner on that topic, but we're using a buildbot. It's still experimental because running an arbitrary setup.py can fail for many reasons. Another thing: once PEP 345 has the required changes (having metadata fields with platform conditions) we will be able to do some checks without having to run any code for any field located in PKG-INFO In any case, I am still not convinced that these checks should be forced on PyPI side when the sdist is uploaded. I see this as a QA rating, because even if a project's setup.py is great, other things can be wrong in the project's code itself. Tarek On 11/9/09, Sridhar Ratnakumar wrote: > On Sat, 07 Nov 2009 07:37:37 -0800, Tarek Ziadé > wrote: > >>> >>> The solution for a better PyPI: >>> >>> - more checks, more restrictions >>> - every package maintainer uploading something to PyPI >>> should have a certain attitude that PyPI is a public >>> resource where the content should met certain >>> quality criteria and where each package has >>> a certain responsibility to Python community. >> More checks would be nice, so we can provide QA rates or something >> similar. >> I don't think we should enforce any policy whatsoever though at PyPI. >> We can't force people that upload distributions to >> comply with some strict QA rules imho (no binary distro allowed if no >> sdist is present for example). > > I suggest that we check for valid metadata on the uploaded sdists at the > least. If you visit http://pypm.activestate.com/ - most failed packages > are due to the fact the sdist uploaded by the author misses certain files > such as README.txt (that is read by setup.py) or setup.py/PKG-INFO itself. > > Without such quality policing, I can't see how tools like pip/easy_install > could even install the package (let alone doing it in an user-friendly > way). > > -srid > -- Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Mon, 09 Nov 2009 13:32:55 -0800, "Sridhar Ratnakumar" wrote: > I suggest that we check for valid metadata on the uploaded sdists at the > least. If you visit http://pypm.activestate.com/ - most failed packages > are due to the fact the sdist uploaded by the author misses certain files > such as README.txt (that is read by setup.py) or setup.py/PKG-INFO itself. That may be out of the scope of distutils. Not sure. CPAN does do checks and we've taken it upon ourselves to start writing something similar. Our target is to set up to run daily and package validate run on a server farm. It's located at : http://bitbucket.org/djlyon/pypi-package-testbot/ > Without such quality policing, I can't see how tools like pip/easy_install > could even install the package (let alone doing it in an user-friendly > way). It's a community issue, sure. And it affects everyone when something breaks because of some very minor fault. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, 07 Nov 2009 07:37:37 -0800, Tarek Ziadé wrote: The solution for a better PyPI: - more checks, more restrictions - every package maintainer uploading something to PyPI should have a certain attitude that PyPI is a public resource where the content should met certain quality criteria and where each package has a certain responsibility to Python community. More checks would be nice, so we can provide QA rates or something similar. I don't think we should enforce any policy whatsoever though at PyPI. We can't force people that upload distributions to comply with some strict QA rules imho (no binary distro allowed if no sdist is present for example). I suggest that we check for valid metadata on the uploaded sdists at the least. If you visit http://pypm.activestate.com/ - most failed packages are due to the fact the sdist uploaded by the author misses certain files such as README.txt (that is read by setup.py) or setup.py/PKG-INFO itself. Without such quality policing, I can't see how tools like pip/easy_install could even install the package (let alone doing it in an user-friendly way). -srid ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Ian Bicking wrote: > I don't think changing distutils to improve error output would be hard > at all. It looks like there's a line in distutils.core that catches > these exceptions (and doesn't look like it actually catches all > exceptions?), and that can just be fixed. I agree this is one of the thing which can be improved without unintended consequences. > > Another topic that has come up: I do agree subclassing makes it really > hard to have multiple lines of development (specifically something > like Setuptools or Distribute, along with ad hoc development in > setup.py files). But I think that can be resolved. Perhaps, for > instance, Distribute can have implementations of commands like build, > that can easily be customized or extended without subclassing (e.g., > by pre-build or post-build functions). In numpy's case, we subclass the Distribution class mainly to add new data which are shared within commands (for example, to build pure C libraries made available to 3rd parties, or to keep track of the scons scripts). I have myself tried the pre/post hooks for commands (for numpy.distutils), but I did not go very far: the problem was almost always linked to commands which need to share knowledge from each other. OTOH, I tried this when I started pocking into numpy.distutils 2 years ago, maybe I could go further today. Some things are fixable in distutils: for example, to build things, you should be able to get rid of the imperative operations, and have instead of registry of extension -> action (ala scons/waf). This would make adding new tools (cython, assembly, etc...) easier, as you would add tools through a registry instead of overriding build_ext as currently done by e.g. numpy.distutils or cython.distutils. Doing so while keeping backward compatibility would be hard, though. > I'd really be shocked if a > rewrite of distutils was necessary, or even necessary to simplify > things. That's the opposite of my own experience. I think I have given several reasonable examples of shortcomings of distutils: I would be glad to hear that each of them has a simple solution which is backward-compatible in distutils. But another way to look at it is to ask: What is there in distutils that you would consider important to keep ? It is important that people who maintain packages do not have to rewrite their setup to a new hypothetical system: asking thousand of developers to rewrite their setup would be insane. But this can be done without being tied to distutils API. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Mon, Nov 9, 2009 at 1:09 AM, Ben Finney wrote: > David Lyon writes: > >> On Sun, 8 Nov 2009 23:28:42 -0600, Ian Bicking wrote: >> >> > In the tools I've written, I generally give the traceback if the >> > verbosity is turned up, and in a case like this (an unexpected >> > exception -- for distutils that's any exception except a few that >> > distutils defines) I include "use -v for more". >> > >> > I think distutils (or Distribute) could do the same. >> >> Hi Ian, >> >> I think PIP is quite an accomplishment. >> >> But don't you think that its a big ask to refactor >> distutils/distribute to redo their error messages for package >> building? > > I've just had a read through the code for ‘pip’; AFAICT, the “redo the > error messages for package building” essentially amounts to using the > ‘logging’ module. Is that a big ask? pip doesn't use the logging module, it uses its own logger, which is intended more for managing the output of a command-line program and not just post-mortem debugging. I don't think changing distutils to improve error output would be hard at all. It looks like there's a line in distutils.core that catches these exceptions (and doesn't look like it actually catches all exceptions?), and that can just be fixed. Another topic that has come up: I do agree subclassing makes it really hard to have multiple lines of development (specifically something like Setuptools or Distribute, along with ad hoc development in setup.py files). But I think that can be resolved. Perhaps, for instance, Distribute can have implementations of commands like build, that can easily be customized or extended without subclassing (e.g., by pre-build or post-build functions). I'd really be shocked if a rewrite of distutils was necessary, or even necessary to simplify things. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
11/09/2009 09:57 AM, Ben Finney: For example, debian packaging what is on pypi must be straighforward. (same for RPM based distribution and other systems) Packaging for Debian is much more about following the policy, which deliberately involves human intervention and judgement, and can't very well be automated. As far as I know (I might know nothing;)), and as far as I follow the debian-mentors ML, most of the "problems" is about filesystem hierarchy Python packages are pretty clear on their FS hierarchy, normalizing it should not be that hard. Once it's clear about that, Debian packaging straighforwardness is 80% done. Can you give an example of what you mean: * How straightforward do you find performing the Debian packaging for a Perl package, and what tools do you use to do it? * Would you consider it sufficient for the same (or equivalent) process to apply for Debian packaging of a Python package? Give me just some time to give a try on an examples. My tries might invalidate my assumptions, but I need to test that on a bunch of Perl modules to confirm. Not just on 2-3. -- Architecte Informatique chez Blueline/Gulfsat: Administration Systeme, Recherche & Developpement +261 33 11 207 36 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Ben Finney wrote: > Rakotomandimby Mihamina writes: > > >> For example, debian packaging what is on pypi must be straighforward. >> (same for RPM based distribution and other systems) >> > > Packaging for Debian is much more about following the policy, which > deliberately involves human intervention and judgement, and can't very > well be automated. RPM, as I understand it, is more lax and a simple RPM > can be generated quite automatically. > A .deb can also be automated up to a certain level. Even for packages like numpy, there is not much needed. Also, the official debian policy is mandatory for adoption into official repositories - but if I need to push a simple .deb to colleagues for a new package of mine, I don't want/need to spend that much time. For example, if distutils were to support the --*dir options of autoconf plus all the related metadata to tag files accordingly, it would make the task quite simple. Even for complex package, this would make packaging and package update simpler for package maintainer. If I look at the debian dir for the numpy package (in Ubuntu), there is really not much needed: the .install files to tag which files to install for which package, adding debian specific files (README.debian and co). Most of it if not all could be removed with the corresponding distutils support (which would not be .deb specific in any way, and would help rpm, .pkg and even windows packaging as well though .e.g. nsis files). David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
David Lyon writes: > On Sun, 8 Nov 2009 23:28:42 -0600, Ian Bicking wrote: > > > In the tools I've written, I generally give the traceback if the > > verbosity is turned up, and in a case like this (an unexpected > > exception -- for distutils that's any exception except a few that > > distutils defines) I include "use -v for more". > > > > I think distutils (or Distribute) could do the same. > > Hi Ian, > > I think PIP is quite an accomplishment. > > But don't you think that its a big ask to refactor > distutils/distribute to redo their error messages for package > building? I've just had a read through the code for ‘pip’; AFAICT, the “redo the error messages for package building” essentially amounts to using the ‘logging’ module. Is that a big ask? -- \“We have to go forth and crush every world view that doesn't | `\believe in tolerance and free speech.” —David Brin | _o__) | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Rakotomandimby Mihamina writes: > For example, debian packaging what is on pypi must be straighforward. > (same for RPM based distribution and other systems) Packaging for Debian is much more about following the policy, which deliberately involves human intervention and judgement, and can't very well be automated. RPM, as I understand it, is more lax and a simple RPM can be generated quite automatically. Can you give an example of what you mean: * How straightforward do you find performing the Debian packaging for a Perl package, and what tools do you use to do it? * Would you consider it sufficient for the same (or equivalent) process to apply for Debian packaging of a Python package? -- \ “If consumers even know there's a DRM, what it is, and how it | `\ works, we've already failed.” —Peter Lee, Disney corporation, | _o__) 2005 | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
11/06/2009 08:53 PM, Guido van Rossum: Python is suffering a lot in the scientific word, because it has not a CPAN-like repository. One thing I think is important too: Perl has "helpers" that help to distribution-package easily. Python must have too. For example, debian packaging what is on pypi must be straighforward. (same for RPM based distribution and other systems) -- Architecte Informatique chez Blueline/Gulfsat: Administration Systeme, Recherche & Developpement +261 33 11 207 36 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sun, 8 Nov 2009 23:28:42 -0600, Ian Bicking wrote: > In the tools I've written, I generally give the traceback if the > verbosity is turned up, and in a case like this (an unexpected > exception -- for distutils that's any exception except a few that > distutils defines) I include "use -v for more". > > I think distutils (or Distribute) could do the same. Hi Ian, I think PIP is quite an accomplishment. But don't you think that its a big ask to refactor distutils/distribute to redo their error messages for package building? imo the basic problem in setup.py and package building comes from ambiguity in the file specification section. Everything else is pretty much ok. With the declarative format David C is hinting at (from Haskell) it should be a lot easier to collect those files up and put them in a .tar.gz archive. I'm just wondering if you've ever thought about the tool chain on the other side from pip. Like how hard is it to create archive files of source with all the right stuff. So I guess what's your take on how close to being right is David C? Best Regards David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sun, Nov 8, 2009 at 10:28 PM, Robert Kern wrote: > David Cournapeau wrote: > >> I don't understand what's there to buy. Several people reported >> distutils errors without any backtrace, though a fair shared of those >> were caused by our own extensions. > > distutils specifically swallows exceptions and formats them for users by > default. After all, it is trying to behave like a regular command line > program that interacts with users, not developers. This is easily > overridable for developers who are trying debug problems by setting the > environment variable DISTUTILS_DEBUG=1. This will make distutils just give > the traceback. In the tools I've written, I generally give the traceback if the verbosity is turned up, and in a case like this (an unexpected exception -- for distutils that's any exception except a few that distutils defines) I include "use -v for more". I think distutils (or Distribute) could do the same. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Robert Kern wrote: > After all, it is trying to behave like a regular command line program > that interacts with users, not developers. This is easily overridable > for developers who are trying debug problems by setting the > environment variable DISTUTILS_DEBUG=1. This will make distutils just > give the traceback. I did not know about that option, thanks. The last example I remember was python 2.6 breaking our mingw support: the raised exception was an empty message, so you got a "error:" as your error message when the exception was caught in setup. I am not sure it is better than a traceback: maybe distutils could separate between 'expected' and 'unexpected' exceptions to make this clearer. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
David Cournapeau wrote: I don't understand what's there to buy. Several people reported distutils errors without any backtrace, though a fair shared of those were caused by our own extensions. distutils specifically swallows exceptions and formats them for users by default. After all, it is trying to behave like a regular command line program that interacts with users, not developers. This is easily overridable for developers who are trying debug problems by setting the environment variable DISTUTILS_DEBUG=1. This will make distutils just give the traceback. Is this what you are referring to? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Tarek Ziadé wrote: > On Sun, Nov 8, 2009 at 2:22 PM, David Cournapeau > > [..] >> - if you extend an existing command, you have to take care whether you run >> under setuptools or distutils (and soon distribute will make this worse). > > No, because unlike setuptools, we want to remove the patching that is done on > the Command class and on the DIstribution class, and make Distribute a good > Distutils citizen in that matter. I don't think that's possible, not without major changes of the distutils design at least. Right now, in numpy.distutils, we extend the Distribution class to hold a few additional data. We inherit from setuptools or distutils depending on whether setuptools has been imported. The fundamental problem is not so much that setuptools does monkey-patching, but that you need to inherit to extend. Once you have several possible packages which inherit independently from those classes, you have to special case each of them if you want to support all of them. > > > > This is going to be changed because I am currently refactoring the > build_ext command > so such methods are in a base "compiler" class and eventually in util.py Refactoring cannot help, unless you fundamentally change how it works. I gave an example of a simple feature which is hard to do with current distutils: obtaining the install prefix early in the build scheme. When you do: python setup.py install --prefix=foo the build_* commands don't know prefix, only install knows it. In numpy, I added various hack each more ugly than the other to fake running the install command before running the build commands to obtain this information. I would be surprised if subsequent refactor of distutils will not break it. > I agree the code is not modern, but things are getting changed through > small steps. > > Although, I don't buy the "strange errors" part and things getting swallowed > :) I don't understand what's there to buy. Several people reported distutils errors without any backtrace, though a fair shared of those were caused by our own extensions. Concerning changing into small steps: I understand that changing gradually is often better than starting from scratch, but my own experience with numpy.distutils and distutils convinced me that there is not much to save from distutils code. I consider the API, the UI and the implementation deeply flawed. For me, an "improved backward compatible" distutils is an oxymoron. More practically, I think the recent distutils-related changes in 2.6.3 and 2.6.4 will keep happening. In numpy.distutils, we depend a lot on internal details of distutils, because we need things not available in the API, and I don't think we are alone. I would mention that numpy is one of the package which has the most invested in distutils in terms of code (sloccount tells me that numpy.distutils is 10315 LOC, whereas python 2.6.2 distutils is 10648 LOC), so I am certainly aware of the cost of breaking backward compatibility. > My opinion is that you've build something else when Distutils was not > evolving anymore. It has nothing to do with distutils evolving. We have had our own extensions for years (I am a relatively newcommer, but some numpy.distutils code goes back to 2001), so we could have done pretty much what we wanted. We now have two build systems: one based on distutils, one based on scons. Every-time I added a build feature to numpy, it took me much more time to add it to distutils than to the scons build. The code needed to support scons build is ~ 2000 lines of code, is more robust, handle dependencies automatically, support reliable parallel builds, and is more maintainable. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sun, 8 Nov 2009 19:19:44 +, Floris Bruynooghe wrote: > .. improve the PEP 386 > reference implementation for example (I pick that one as I know most > about it from all the PEP proposals currently). I'm +1 on PEP-386. It makes sense to me. > This is a PEP that should maybe be finished first of the bunch PEP-345 is perphaps more important. More depends on that. I think/hope Guido's post has changed things a little. I agree with those who say everybody should work together to some PEPs closed off and I also agree with the push to offer something that is of a more comparable standard to CPAN. To get to where Guido is asking for, I think there are some gaps in the PEP coverage. So we need to cover those bases also. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, Nov 07, 2009 at 10:56:24AM -0500, David Lyon wrote: > On Sat, 7 Nov 2009 16:08:46 +0100, Tarek Ziadé > wrote: > > > Gosh. I am not your boss, and I am not telling you what to do. > > otoh you're the boss of distutils. So you can direct people to > work on certain things to help you along. That would have kept > me much quieter with work. If you have time to burn and need to be told how to spend it: on top of my head a useful contribution would be to improve the PEP 386 reference implementation for example (I pick that one as I know most about it from all the PEP proposals currently). This is a PEP that should maybe be finished first of the bunch, so that's definitely useful. Useful things: * Read up on the PEP and all the documentation in the reference implementation * Check out previous discussions and make sure nothing is missed. * Add more test! The core class could be tested better, and suggest_rational_version is massively undertested. For this you need to read up on the old distutils version scheme as well as setuptools scheme and build lots of test cases for it. Also looking at all versions on the cheesshop to figure out tricky version numbers and test the function. All the "reasonable" version numbers used on PyPI should be tested really. * Check the history of the implementation. Maybe there have been checkins that correct things in one area but left similar bugs in other areas (e.g. a bug fixed in the class but not in suggest_rational_version). If so add tests. * If you managed to create tests that fail, see if you can figure out ways to make them pass. * Improve the documentation of the reference implementation, after you've done many of the above things you'll definitely have found some places where it could be improved. * Create patches out of the above work and submit it to the refrence implementation, if they're useful and good they'll get accepted. But don't be discouraged if a re-work is asked for initially. I'm not telling you the above things because I want to be your (or anyone's) boss and tell other people what to do. I'm telling you them as an example of how to contribute to the current work. The important thing is that I don't know anything more then you do, I haven't had secret conversations with a cabal or so (I'm not even on IRC usually). The only thing I do is read the mailing list and look at proposals I'm interested in. Finding and carrying out work like this is contributing. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sun, Nov 8, 2009 at 2:22 PM, David Cournapeau [..] > - if you extend an existing command, you have to take care whether you run > under setuptools or distutils (and soon distribute will make this worse). No, because unlike setuptools, we want to remove the patching that is done on the Command class and on the DIstribution class, and make Distribute a good Distutils citizen in that matter. IOW you won't suffer anymore from what you've described. > Those commands will not work the same when you run them under paver > either. > - the division in subcommands is painful, and the abstraction does not make > much sense IMHO. Recently, I needed to access simple things like library > filename (foo ->libfoo.a/foo.lib/etc..), install prefix. But those > are not > accessible to each command. [...] > - if you want to add a new source file extension, you have to rewrite the > build_ext or build_src command and you often cannot reuse the base class > methods. > - etc... This is going to be changed because I am currently refactoring the build_ext command so such methods are in a base "compiler" class and eventually in util.py You are welcome to help in this refactoring, > Also, the distutils code is horrible: you don't really know what's > public and > what's not, most attributes are added at runtime (and sometimes differ > depending on the platform). > Often, you get strange errors with the exception > swallowed, and that happens only on some platforms for some users; in that > case, the only way to debug it is to be able to run their platform. When you > write extensions to distutils, this contributes to the whole unpleasant > experience. I agree the code is not modern, but things are getting changed through small steps. Although, I don't buy the "strange errors" part and things getting swallowed :) [..] > > Guido wanted to know how scientific python people feel about the whole > situation, and my own impression is that we are going further from what we > need. I don't think anything based on distutils can help us. This is not to > criticize Tarek, PJE and other people's work: I understand that > distutils and > setuptools solve a lot of problems for many people, and I may just be a > minority. My opinion is that you've build something else when Distutils was not evolving anymore. This is not true anymore. It's moving again. And I think that the work that is going on is heading in the right direction, even for your use cases imho. If projects that maintain distutils patched versions push those patches now in the Python issue tracker, diffing against the current trunk, and if those patches make sense and are with tests, that's by far the *easiest* way to help improving Distutils. And that's the easiest work for me : I'll just review them and commit them if they do improve Distutils. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Georg Brandl wrote: > > One thing about CPAN (and Haskell's libraries on hackage) that I think > many people see favorably, even though it's only superficial, is the > more-or-less consistent hierarchical naming for the individual packages > (or the contained modules in Haskell). Compared with that, the Python > package namespace looks untidy. That's true, but there is not much we can do on this one, so I did not mention it. > > Note that the downloadable distutils manual has 94 pages and *should* be > enough to explain the basics of packaging. It has to be updated, of > course, once the more advanced mechanisms are part of the core. The manual is too complicated for simple tasks, and not very useful for complex ones. Mostly because distutils does not follow the "only one way to do things" mantra. I can help to improve the distutils documentation for the build part, which is mostly undocumented (things like how to create a new command to build ctypes extensions, etc...). > > Me too. Though it would be Snakebite + serious sandboxing. Sandboxing is of course needed, but that's a known problem, and people have already thought hard about it. The open suse build system, albeit linux specific, works quite well, for example. For environment sandboxing, chroot works on all unix I know (including mac os x) - security is more challenging, I don't have any expertise there. Windows is more difficult to handle, though (maybe windows people know good sandboxing solutions outside full-blown vm). > What you're saying there about Cabal is exactly my experience. It is very > nice to work with, and I've not yet seen a conceptual failure. > > But we're not far from that, with a static metadata file. Several people have claimed this so far, but I don't understand why - could you expand on this ? My impression is that the focus is mostly on version specification and install/build requirements in the static data, but to me that's a tiny detail. I want something like .cabal files, where you can specify documentation, data, source files, etc... Something like what I started to prototype there: http://github.com/cournape/toydist/ To take an example you are well familiar with, you can fully describe sphinx with it, and the conversion is mostly automatic. This is not even 500 LOC. With this kind of design, you can use different build systems on top of it (there is for example unpublished code in toydist to use a scons-based build system instead of distutils as currently done). > >> I won't rehearse it here, but basically: >> - distutils is too complex for simple packages, and too inflexible >> for complex ones. Adding new features to distutils is a painful >> experience. Even autotools with its mix of 100 000 lines autogenerated >> shell code, perl, m4 is more pleasant. > > Really? Sure, the perl/shell/awk/m4 mix is painful, but at least the result is reasonably robust, and can be extended. > I would have assumed that even writing a whole new distutils > command/build step can't be more painful than adding the equivalent to > an autotools-style build system, being Python after all. However, I've > never done such a thing, so I have to believe you. I expand on that, because I think few people understand the problem here, and that's maybe the main source of frustration for core numpy developers as far as distutils is concerned. True, writing your own command is easy. But it has many failure modes: - if you extend an existing command, you have to take care whether you run under setuptools or distutils (and soon distribute will make this worse). Those commands will not work the same when you run them under paver either. - the division in subcommands is painful, and the abstraction does not make much sense IMHO. Recently, I needed to access simple things like library filename (foo ->libfoo.a/foo.lib/etc..), install prefix. But those are not accessible to each command. The install prefix was particularly painful, it took me several hours to get it work right with distutils inplace, develop mode on all platforms. All this is trvially easy to get with autotools, scons or waf. Every new feature I needed to add to numpy.distutils was an unpleasant experience. I had to read the distutils sources (for every supported python version), run it on several platforms, and got it working by trial an error. - if you want to add a new source file extension, you have to rewrite the build_ext or build_src command and you often cannot reuse the base class methods. - etc... Also, the distutils code is horrible: you don't really know what's public and what's not, most attributes are added at runtime (and sometimes differ depending on the platform). Often, you get strange errors with the exception swallowed, and that happens only on some platforms for some users; in that case, the only way to debug it is to be able to run their platform. When you write extensions to distutils, this contributes
Re: [Distutils] People want CPAN :-)
On Nov 7, 2009, at 7:10 PM, Kevin Teague wrote: Being able to update individual modules within the standard library would absolutely rock my world and would have removed the urgency from 2.6.3/4 updates. Just update what's necessary. What a concept. Yes, please. The sooner the better. As long as the testing is at least as thorough, and reverting is very easy, why _wouldn't_ this be a good idea? I gotta say, all this talk of CPAN reminds me of when I first started using Python and was mystified/mortified at the total lack of something similar. I kind of got used to it but this is bringing back the itch in a major way... S ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Nov 7, 2009, at 3:32 PM, Jannis Leidel wrote: Oh, intriguing idea, has moving distutils out of Python core been considered before? To be clear, I'm not suggesting moving anything in or out of the standard library. Just taking what's in the standard library and packaging it up, and allowing for releases of these packages to happen on PyPI (or for people to more easily include VCS checkouts of these packages in their Buildouts or Pip-outs). The main Python interpreter download would still include the same standard library packages (or whatever the python-dev wants to be in there), and they would still be installed in the same way, just with the addition of .egg-info files to make them PEP 376 compliant (right now wsgiref in the standard library has an .egg-info but it's an exception). It's been considered before (Chris Withers was recently asking for it), and perhaps before that. I was kvetching about this on the stdlib- sig last month, where I outlined some of the other problems that not having standard packaging for the standard library presents: http://mail.python.org/pipermail/stdlib-sig/2009-October/000721.html ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Am 08.11.2009 um 00:18 schrieb Kevin Teague: One other really nice thing about the Perl packaging ecosystem is that their standard library is packaged! If there is a bug found in the Perl standard library, it's trivial to upgrade it to a newer release with a bug fix. For example, the recent little distutils snafu would have been a lot less painful for the end user's if we'd been able to get the bug fix with a simple: $ pip install --upgrade distutils In this respect, from an end user perspective, it really feels like you're getting hit with a stick, "Need that distutils fix? Hmm, well, OK, but you're going to have wait another month until we do a full Python release, and then accept all these other unrelated changes if you want that ... " CPAN even informs you if there's a newer release of itself available, and suggests you might like to upgrade: There's a new CPAN.pm version (v1.9402) available! [Current version is v1.7602] You might want to try install Bundle::CPAN reload cpan without quitting the current session. It should be a seamless upgrade while we are running... Buildout has a similar upgrade notification feature, but I don't think pip or easy_install does? One nice thing about setuptools (or now distribute) not having been merged into the standard library is that they're easy to update to newer releases. As we push more of this code down into Distutils, we are making it more difficult to get updates :( Oh, intriguing idea, has moving distutils out of Python core been considered before? Jannis ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
One other really nice thing about the Perl packaging ecosystem is that their standard library is packaged! If there is a bug found in the Perl standard library, it's trivial to upgrade it to a newer release with a bug fix. For example, the recent little distutils snafu would have been a lot less painful for the end user's if we'd been able to get the bug fix with a simple: $ pip install --upgrade distutils In this respect, from an end user perspective, it really feels like you're getting hit with a stick, "Need that distutils fix? Hmm, well, OK, but you're going to have wait another month until we do a full Python release, and then accept all these other unrelated changes if you want that ... " CPAN even informs you if there's a newer release of itself available, and suggests you might like to upgrade: There's a new CPAN.pm version (v1.9402) available! [Current version is v1.7602] You might want to try install Bundle::CPAN reload cpan without quitting the current session. It should be a seamless upgrade while we are running... Buildout has a similar upgrade notification feature, but I don't think pip or easy_install does? One nice thing about setuptools (or now distribute) not having been merged into the standard library is that they're easy to update to newer releases. As we push more of this code down into Distutils, we are making it more difficult to get updates :( ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Nov 7, 2009, at 6:57 AM, Andreas Jung wrote: - no more external hosting of packages. If people want their packages listed on Pypi, they should be required to upload their packages on PyPI (no more issues with non-available external server, no more issues with mirroring external servers, no more issues with wrong download URLs within package metadata) Amen! Externally hosted packages are a royal PITA and it makes it so much harder to automatically install dependencies. This field should at the least be marked in the Distutils docs with a big "DEPRECATED: DO NOT USE" note. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Am 07.11.09 17:13, schrieb Tarek Ziadé: > On Sat, Nov 7, 2009 at 4:56 PM, Andreas Jung wrote: > [..] > >> Do we need/want development on PyPI? At least not me. >> >> MAJOR.MINOR.MICRO.PICO + |a-c]1..N >> >> should be good enough. >> > PEP 386 is about providing the version scheme so we can compare versions > in Distutils when we want to know if a dependency is met (like what > setuptools does). > > So its wider than PyPI : people need to be able to compare development > versions as well. > So for example, zc.buildout can rely on it for your daily work. > ACK for a more necessity of one more complex versioning schema in general (but we don't need to support all variants) but we don't need to support dev packages on PyPI - that's why a stronger version check should be enforced. > [...] > >> "community" does not imply that we can not agree on certain rules and >> standards >> for PyPI - otherwise PyPI remains as it sometimes appears - an unflashed >> package toilet. Python as a quality programming language needs a package >> repository with some minimum standards - I completely disagree with >> "community" >> as a synonym for "we must make everyone happy". >> >> > But the philosophy of Python is to provide a multi-paradigm language I think, > without forcing any strong rule like this. (unlike Java I guess) > > My mother (sorry that's the example I have in my mind) is using Python > in her university > math /statistics lab, and they don't really care about QA. > > But she might push her software at PyPI one day. She won't if its > rejected because > she doesn't follow a version scheme, or push a binary release rather > than a source one. > > I think your mother (and most others) are smart enough to understand and support a simple versioning schema. Bringing it to the point in different way: "community" does not mean anarchy. Package maintainers have a lot of freedom but as said also a responsibility for their software - otherwise redeclare PyPI as package t* (I mentioned the word already) > Its good too have industrial-strength conventions, so we can build > industrial-level applications, > but I think we need to be careful about the ticket entry for PyPI. > > Wouldn't be better to set these enforcements in subcommunity like > plone.org where it would > make a lot of sense to enforce QA for plone packages ? > (plone.org has PyPI support) > I don't care about subcommunities at this point. PyPI is a central resource to Python. It is essentional for my daily work. It is essential for me that the packages having reasonable metadata. It is essential for me that the packages are available all the time. A certain quality and standards are especially essential to non-professional Python users and developers - nothing is more frustrating for those people than dealing with non-functional packages, undocumented packages or packages of pre-alpha quality. Andreas <>___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Nov 7, 2009, at 12:39 AM, Lennart Regebro wrote: 2009/11/7 Kaelin Colclasure : Since I bootstrapped the environment I used to learn Python with easy_install, I naturally went straight to the easy_install docs to learn how to give back. I wasn't even aware of this separate thing called distutils until I read about it in the easy_install (err setuptools) documentation (sic). Yes, this seems to be a reasonably way to realize that packaging is done with distutils. What was then complicated with the distutils docs? It's so long ago I did this the first time I don't remember if I found it difficult. The setuptools docs I read left me with the impression that distutils was more about building C extensions and that if my package was pure Python source (which it was) I should not need anything more than setuptools. And this did prove true, eventually -- but only after I restructured my source into a package to be more like setuptools wanted it. What wasted quite a bit of time along the way was that I found other examples of setup.py files that were using setuptools but were also using some underlying distutils conventions for packaging a single module. This approach wasn't mentioned in the setuptools docs but it *looked* more like what I was trying to do. But then, it turned out there was no simple way to use the single-module spec *and* get my static data installed along with the code. HTH, -- Kaelin -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, Nov 7, 2009 at 4:56 PM, Andreas Jung wrote: [..] > > Do we need/want development on PyPI? At least not me. > > MAJOR.MINOR.MICRO.PICO + |a-c]1..N > > should be good enough. PEP 386 is about providing the version scheme so we can compare versions in Distutils when we want to know if a dependency is met (like what setuptools does). So its wider than PyPI : people need to be able to compare development versions as well. So for example, zc.buildout can rely on it for your daily work. [...] > > "community" does not imply that we can not agree on certain rules and > standards > for PyPI - otherwise PyPI remains as it sometimes appears - an unflashed > package toilet. Python as a quality programming language needs a package > repository with some minimum standards - I completely disagree with > "community" > as a synonym for "we must make everyone happy". > But the philosophy of Python is to provide a multi-paradigm language I think, without forcing any strong rule like this. (unlike Java I guess) My mother (sorry that's the example I have in my mind) is using Python in her university math /statistics lab, and they don't really care about QA. But she might push her software at PyPI one day. She won't if its rejected because she doesn't follow a version scheme, or push a binary release rather than a source one. Its good too have industrial-strength conventions, so we can build industrial-level applications, but I think we need to be careful about the ticket entry for PyPI. Wouldn't be better to set these enforcements in subcommunity like plone.org where it would make a lot of sense to enforce QA for plone packages ? (plone.org has PyPI support) Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On 03:56 pm, li...@zopyx.com wrote: Am 07.11.09 16:37, schrieb Tarek Ziad�: On Sat, Nov 7, 2009 at 3:57 PM, Andreas Jung wrote: [..] - supports too much different versioning schemas. Both schema supported by setuptools and the one proposed by Tarek in some PEP are totally over-engineered. A simple and *enforced* versioning schema is what I want to see. Unfortunately, as long as we have release candidates, and development versions, we need a more complex scheme that MAJOR.MINOR.MICRO. Do we need/want development on PyPI? At least not me. MAJOR.MINOR.MICRO.PICO + |a-c]1..N should be good enough. Please be considerate of the time of other people and read the previous threads on this list about versioning schemes before proposing a new one. If you have done this, then please make it evident by describing how your proposed scheme addresses the problems raised in previous discussions, or how those problems are not important/real/whatever. Thanks. Jean-Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, 7 Nov 2009 16:08:46 +0100, Tarek Ziadé wrote: > Gosh. I am not your boss, and I am not telling you what to do. otoh you're the boss of distutils. So you can direct people to work on certain things to help you along. That would have kept me much quieter with work. > I am just telling you that if you want to help in the static metadata > field, and if your goal is to help *Distutils* improve, you have to > work with what has been started! Well talk about static metadata installation was started at pycon last year. Nothing happened till I started working on it. And I believe that I'm the only one working on it now. > If you work on a PEP that is related to any PEP started in the same > area, I will strongly oppose against adding your PEP > because it is a non-sense. Same area? as in any packaging PEP? any distutils PEP? There are PEPs in every area. And PEPs are inter-related. Does rejection of PEPs on the above grounds apply to everyone? or just me? Other people seem to have PEPs in related areas. > We don't add new PEPs for the heck of it. And people don't write them for the heck of it either. If I see something I can do to assist existing PEPs I'll try to do so. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Am 07.11.09 16:37, schrieb Tarek Ziadé: > On Sat, Nov 7, 2009 at 3:57 PM, Andreas Jung wrote: > [..] > >> - supports too much different versioning schemas. Both >> schema supported by setuptools and the one proposed >> by Tarek in some PEP are totally over-engineered. >> A simple and *enforced* versioning schema is what >> I want to see. >> > Unfortunately, as long as we have release candidates, and development > versions, > we need a more complex scheme that MAJOR.MINOR.MICRO. > > Do we need/want development on PyPI? At least not me. MAJOR.MINOR.MICRO.PICO + |a-c]1..N should be good enough. > Last, we can encourage people to use it, but we can't enforced it: > Of course we can.. > I know people that are happily using dates for their versions, and we > can't forbid > them to push their work on pypi just because of that. > ..one must not accept and support a whole zoo of private numbering schemes. Agree on a common and minimal standard and enforce the standard. > We can try to educate then, but that's their pick at the end I think. > Teaching is a good thing... > An enterprise PyPI could do enforce it, but not our community PyPI imho "community" does not imply that we can not agree on certain rules and standards for PyPI - otherwise PyPI remains as it sometimes appears - an unflashed package toilet. Python as a quality programming language needs a package repository with some minimum standards - I completely disagree with "community" as a synonym for "we must make everyone happy". Andreas <>___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
At 03:57 PM 11/7/2009 +0100, Andreas Jung wrote: - supports too much different versioning schemas. Both schema supported by setuptools and the one proposed by Tarek in some PEP are totally over-engineered. A simple and *enforced* versioning schema is what I want to see. - no more external hosting of packages. If people want their packages listed on Pypi, they should be required to upload their packages on PyPI (no more issues with non-available external server, no more issues with mirroring external servers, no more issues with wrong download URLs within package metadata) Do note that at least these two requirements of yours are likely to be opposed by some with at least as much force (if not more so) than you are proposing them with. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, Nov 7, 2009 at 3:57 PM, Andreas Jung wrote: [..] > - supports too much different versioning schemas. Both > schema supported by setuptools and the one proposed > by Tarek in some PEP are totally over-engineered. > A simple and *enforced* versioning schema is what > I want to see. Unfortunately, as long as we have release candidates, and development versions, we need a more complex scheme that MAJOR.MINOR.MICRO. If you think of any ways to simplify PEP 386, please help us, Last, we can encourage people to use it, but we can't enforced it: I know people that are happily using dates for their versions, and we can't forbid them to push their work on pypi just because of that. We can try to educate then, but that's their pick at the end I think. An enterprise PyPI could do enforce it, but not our community PyPI imho [...] > > The solution for a better PyPI: > > - more checks, more restrictions > - every package maintainer uploading something to PyPI > should have a certain attitude that PyPI is a public > resource where the content should met certain > quality criteria and where each package has > a certain responsibility to Python community. More checks would be nice, so we can provide QA rates or something similar. I don't think we should enforce any policy whatsoever though at PyPI. We can't force people that upload distributions to comply with some strict QA rules imho (no binary distro allowed if no sdist is present for example). Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
Andreas Jung kirjoitti: Am 06.11.09 18:53, schrieb Guido van Rossum: I just found this comment on my blog. People have told me this in person too, so I believe it is real pain (even if the solution may be elusive and the suggested solutions may not work). But I don't know how to improve the world. Is the work on distutils-sig going to be enough? Or do we need some other kind of work in addition? Do we need more than PyPI? My 2 cents after reading and ignoring the whole thread: - PyPI provides a good functionality so far What is annoying about PyPI: - some package maintainers have a certain ignorance and arrogance by misusing PyPI - for uploading packages without or broken metadata - for uploading packages of doubtful quality - for uploading packages to PyPI as a replacement for a private egg server - supports too much different versioning schemas. Both schema supported by setuptools and the one proposed by Tarek in some PEP are totally over-engineered. A simple and *enforced* versioning schema is what I want to see. - no more external hosting of packages. If people want their packages listed on Pypi, they should be required to upload their packages on PyPI (no more issues with non-available external server, no more issues with mirroring external servers, no more issues with wrong download URLs within package metadata) - better checks on uploaded packages. A source code release should be made using the 'sdist' command. We don't need source eggs of a package for Python 2.4-2.6 containing Python source code only. +5 The solution for a better PyPI: - more checks, more restrictions - every package maintainer uploading something to PyPI should have a certain attitude that PyPI is a public resource where the content should met certain quality criteria and where each package has a certain responsibility to Python community. +2 Andreas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sat, Nov 7, 2009 at 3:46 PM, David Lyon wrote: [..] > >> .. it means that we can take >> more help from more people (and that includes you of course). > > I really do accept that. > > What's hard for me to understand exactly is what you will actually allow > me to do. Gosh. I am not your boss, and I am not telling you what to do. This is open source, but this is also a community... You can't just come around and ignore some parts of what is being done. I am just telling you that if you want to help in the static metadata field, and if your goal is to help *Distutils* improve, you have to work with what has been started! [..] >> You have started some kind of "counter-PEP" to push another build >> system, and I really doubt this is useful in any ways. > > You shouldn't take it personally that another person tried to submit > a PEP. So many people helped me with my proposal on the mailing > list - and I owe it to them to at least get their efforts taken > into consideration as a PEP. > > In any case, if you feel strongly about this then I can revise > my proposal so that it doesn't look like a proposal for a build > system (it was for a metadata setup - not a build system). [..] > The implication there is you don't want me to submit PEPs because > they somehow counter your work. Well if that's the case I can > be more careful in the future. There's plenty of things > that I can write PEPs for now... Neither Guido, neither Brett neither I are taking this personally. But at some point, if your goal is to help improving Distutils, you have to work with what has been started. > A metadata install was just the first.. > > And to my knowledge, there still isn't a PEP about that. Right? If you work on a PEP that is related to any PEP started in the same area, I will strongly oppose against adding your PEP because it is a non-sense. Take and read the existing PEPs and help us improving them. We don't add new PEPs for the heck of it. If it's a credit issue, your name will be added in any PEP you will provide a valuable help on. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
