On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield <ren...@gmail.com> wrote: > Hi, > > In the toydist proposal/release notes, I would address 'what does > toydist do better' more explicitly. > > > > **** A big problem for science users is that numpy does not work with > pypi + (easy_install, buildout or pip) and python 2.6. **** > > > > Working with the rest of the python community as much as possible is > likely a good goal.
Yes, but it is hopeless. Most of what is being discussed on distutils-sig is useless for us, and what matters is ignored at best. I think most people on distutils-sig are misguided, and I don't think the community is representative of people concerned with packaging anyway - most of the participants seem to be around web development, and are mostly dismissive of other's concerns (OS packagers, etc...). I want to note that I am not starting this out of thin air - I know most of distutils code very well, I have been the mostly sole maintainer of numpy.distutils for 2 years now. I have written extensive distutils extensions, in particular numscons which is able to fully build numpy, scipy and matplotlib on every platform that matters. Simply put, distutils code is horrible (this is an objective fact) and flawed beyond repair (this is more controversial). IMHO, it has almost no useful feature, except being standard. If you want a more detailed explanation of why I think distutils and all tools on top are deeply flawed, you can look here: http://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/ > numpy used to work with buildout in python2.5, but not with 2.6. > buildout lets other team members get up to speed with a project by > running one command. It installs things in the local directory, not > system wide. So you can have different dependencies per project. I don't think it is a very useful feature, honestly. It seems to me that they created a huge infrastructure to split packages into tiny pieces, and then try to get them back together, imaganing that multiple installed versions is a replacement for backward compatibility. Anyone with extensive packaging experience knows that's a deeply flawed model in general. > Plenty of good work is going on with python packaging. That's the opposite of my experience. What I care about is: - tools which are hackable and easily extensible - robust install/uninstall - real, DAG-based build system - explicit and repeatability None of this is supported by the tools, and the current directions go even further away. When I have to explain at length why the command-based design of distutils is a nightmare to work with, I don't feel very confident that the current maintainers are aware of the issues, for example. It shows that they never had to extend distutils much. > > There are build farms for windows packages and OSX uploaded to pypi. > Start uploading pre releases to pypi, and you get these for free (once > you make numpy compile out of the box on those compile farms). There > are compile farms for other OSes too... like ubuntu/debian, macports > etc. Some distributions even automatically download, compile and > package new releases once they spot a new file on your ftp/web site. I am familiar with some of those systems (PPA and opensuse build service in particular). One of the goal of my proposal is to make it easier to interoperate with those tools. I think Pypi is mostly useless. The lack of enforced metadata is a big no-no IMHO. The fact that Pypi is miles beyond CRAN for example is quite significant. I want CRAN for scientific python, and I don't see Pypi becoming it in the near future. The point of having our own Pypi-like server is that we could do the following: - enforcing metadata - making it easy to extend the service to support our needs > > pypm: http://pypm.activestate.com/list-n.html#numpy It is interesting to note that one of the maintainer of pypm has recently quitted the discussion about Pypi, most likely out of frustration from the other participants. > Documentation projects are being worked on to document, give tutorials > and make python packaging be easier all round. As witnessed by 20 or > so releases on pypi every day(and growing), lots of people are using > the python packaging tools successfully. This does not mean much IMO. Uploading on Pypi is almost required to use virtualenv, buildout, etc.. An interesting metric is not how many packages are uploaded, but how much it is used outside developers. > > I'm not sure making a separate build tool is a good idea. I think > going with the rest of the python community, and improving the tools > there is a better idea. It has been tried, and IMHO has been proved to have failed. You can look at the recent discussion (the one started by Guido in particular). > pps. some notes on toydist itself. > - toydist convert is cool for people converting a setup.py . This > means that most people can try out toydist right away. but what does > it gain these people who convert their setup.py files? Not much ATM, except that it is easier to write a toysetup.info compared to setup.py IMO, and that it supports a simple way to include data files (something which is currently *impossible* to do without writing your own distutils extensions). It has also the ability to build eggs without using setuptools (I consider not using setuptools a feature, given the too many failure modes of this package). The main goals though are to make it easier to build your own tools on top of if, and to integrate with real build systems. > - a toydist convert that generates a setup.py file might be cool :) toydist started like this, actually: you would write a setup.py file which loads the package from toysetup.info, and can be converted to a dict argument to distutils.core.setup. I have not updated it recently, but that's definitely on the TODO list for a first alpha, as it would enable people to benefit from the format, with 100 % backward compatibility with distutils. > - arbitrary code execution happens when building or testing with > toydist. You are right for testing, but wrong for building. As long as the build is entirely driven by toysetup.info, you only have to trust toydist (which is not safe ATM, but that's an implementation detail), and your build tools of course. Obviously, if you have a package which uses an external build tool on top of toysetup.info (as will be required for numpy itself for example), all bets are off. But I think that's a tiny fraction of the interesting packages for scientific computing. Sandboxing is particularly an issue on windows - I don't know a good solution for windows sandboxing, outside of full vms, which are heavy-weights. > - it should be possible to build this toydist functionality as a > distutils/distribute/buildout extension. No, it cannot, at least as far as distutils/distribute are concerned (I know nothing about buildout). Extending distutils is horrible, and fragile in general. Even autotools with its mix of generated sh scripts through m4 and perl is a breeze compared to distutils. > - extending toydist? How are extensions made? there are 175 buildout > packages which extend buildout, and many that extend > distutils/setuptools - so extension of build tools in a necessary > thing. See my answer earlier about interoperation with build tools. cheers, David ------------------------------------------------------------------------------ This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev _______________________________________________ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel