[matplotlib-devel] [PATCH] experimental numscons support in matplotlib
Hi, I don't know if that's of any interest for matplotlib developers, but I added scripts to build matplotlib with numscons: http://github.com/cournape/matplotlib/tree/scons_build Not every configuration is supported yet, but I can successfully build matplotlib on Linux with gtk or wx backends. It only adds 3 files + one configuration example, and does not touch any other file. The advantage of numscons over distutils is automatic dependency handling (no need to rm -rf build to get accurate build), easy compiler flags customization, parallel build, etc... There are some instructions in setupscons.py. It is still experimental (I have not implemented check for QT, as well as windows, macosx and qt backends), but it seems to work well. I will add mac os x and windows backends soon (I started this to debug issues on 64 bits version of matplotlib), cheers, David -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [PATCH] experimental numscons support in matplotlib
Hi Andrew, Andrew Straw wrote: > Dear David, > > It certainly is of interest to me. When I get a little time (maybe this > weekend), I'd like to try it. Specifically, I'd like to setup a buildbot > that would automatically build and run the test suite with it. Along > those lines, is there any reason why it shouldn't work with Ubuntu Hardy > amd64 (8.04) and Python 2.5? Or should I try another distro? (I'll be > setting up a chroot.) > It should work on any distro. I have not tested this really hard yet, though - but it already works better for me than the current setupext.py (I may miss something, but the detection fails horribly on my machine, especially for libs not installed in /usr). There are some configurations which are not supported yet (wxpython < 2.8, tkagg, and win32 + mac os x backends), but it should be relatively easy to add support for it, except maybe for mac os x backend because of objective c because numscons does not handle objective C yet (my focus is windows ATM, but if supporting every configuration is a condition for the patch to be included, I am willing to work on it). > looks pretty unintrusive to me -- I can't see why it would hurt to > include it direct into MPL. The patch could be made smaller and more robust if I were allowed to do some basic refactoring to share configuration data between setup.py and setupscons.py: either fixing setup.py so that it uses a __main__ and does not execute the whole distutils dance at import time, or as I usually do in other projects, putting metadata in a separate file. cheers, David -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [PATCH] experimental numscons support in matplotlib
Andrew Straw wrote: > David Cournapeau wrote: > >> Hi, >> >> I don't know if that's of any interest for matplotlib developers, >> but I added scripts to build matplotlib with numscons: >> >> http://github.com/cournape/matplotlib/tree/scons_build >> >> Not every configuration is supported yet, but I can successfully build >> matplotlib on Linux with gtk or wx backends. It only adds 3 files + one >> configuration example, and does not touch any other file. >> >> The advantage of numscons over distutils is automatic dependency >> handling (no need to rm -rf build to get accurate build), easy compiler >> flags customization, parallel build, etc... There are some instructions >> in setupscons.py. >> >> It is still experimental (I have not implemented check for QT, as well >> as windows, macosx and qt backends), but it seems to work well. I will >> add mac os x and windows backends soon (I started this to debug issues >> on 64 bits version of matplotlib), >> > > Dear David, > > It certainly is of interest to me. When I get a little time (maybe this > weekend), I'd like to try it. I have a question about git as well, actually - I could not update the svn metadata, unfortunately, by using git-svn rebase -l (I used your branch on github and the instructions on matplotlib website). It gives me an awful lot of merging errors, which seems to indicate that git-svn is confused about the current state, cheers, David -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [PATCH] experimental numscons support in matplotlib
Andrew Straw wrote: > Andrew Straw wrote: > >> David Cournapeau wrote: >> >> >>> I have a question about git as well, actually - I could not update the >>> svn metadata, unfortunately, by using git-svn rebase -l (I used your >>> branch on github and the instructions on matplotlib website). It gives >>> me an awful lot of merging errors, which seems to indicate that git-svn >>> is confused about the current state, >>> >>> >> I have trouble with that too. :( >> >> Do you have notes on how you setup the numpy git repo? I was never able >> to figure out the way to make a good git clone that could be shared with >> others. >> > OK, I think I fixed the git mirror. The good news is that things should > work just like your scipy svn git mirror. The bad news is that I moved > the master branch to "old-master-broken-svn-import", so you'll probably > have to rebase all your changes. The new git branch to base off is > "trunk", which is a mirror of the svn trunk. MPL's svn layout doesn't > follow the standard svn repository, so I haven't mirrored other branches > (yet). I'll update the MPL dev docs soon. > FWIW, I have tried importing the whole svn repo, and the repo got might big (~700 Mb) - I guess because of all the things in trunk but not in trunk/matplotlib. > Great > Please let me know if you're having any more trouble. > I have deleted my old repo and redid a fork on github - I have not tried rebasing on top of svn changes, but I updated the scons_build branch anyway. cheers, David -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [PATCH] experimental numscons support in matplotlib
Andrew Straw wrote: > David Cournapeau wrote: > >> Andrew Straw wrote: >> >> >>> Andrew Straw wrote: >>> >>> >>> >>>> David Cournapeau wrote: >>>> >>>> >>>> >>>> >>>>> I have a question about git as well, actually - I could not update the >>>>> svn metadata, unfortunately, by using git-svn rebase -l (I used your >>>>> branch on github and the instructions on matplotlib website). It gives >>>>> me an awful lot of merging errors, which seems to indicate that git-svn >>>>> is confused about the current state, >>>>> >>>>> >>>>> >>>>> >>>> I have trouble with that too. :( >>>> >>>> Do you have notes on how you setup the numpy git repo? I was never able >>>> to figure out the way to make a good git clone that could be shared with >>>> others. >>>> >>>> >>>> >>> OK, I think I fixed the git mirror. The good news is that things should >>> work just like your scipy svn git mirror. The bad news is that I moved >>> the master branch to "old-master-broken-svn-import", so you'll probably >>> have to rebase all your changes. The new git branch to base off is >>> "trunk", which is a mirror of the svn trunk. MPL's svn layout doesn't >>> follow the standard svn repository, so I haven't mirrored other branches >>> (yet). I'll update the MPL dev docs soon. >>> >>> >>> >> FWIW, I have tried importing the whole svn repo, and the repo got might >> big (~700 Mb) - I guess because of all the things in trunk but not in >> trunk/matplotlib. >> >> > OK, I'm rebuilding a repo with the branches and tags myself as we speak. > It's been going over 2 hours on a local rsync on a screaming fast > computer. It took ~ 10 hours on a 4 years-old machine :) > If this does import reasonably, would it be a real pain for > you to rebase again? Not really, I export the branch with format-patch and import with git am, so it takes me less time than writing this very sentence :) >> >> > OK. Actually, the trigger that started all my git work is that something > broke with the old import anyway. Git thought one of the more recent svn > commits was parented to a commit months ago, so everything was broken > anyway. The new import is much better -- for the first time I managed to > be able to share the svn meta data across git clones. > Yes, that's the main point of the numpy/scipy mirrors - Pauli did the work with his git-svn-automirror (which is something you may want to look at if you want an automatic mirror). cheers, David -- Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [PATCH] experimental numscons support in matplotlib
Andrew Straw wrote: > David Cournapeau wrote: > >> Hi, >> >> I don't know if that's of any interest for matplotlib developers, >> but I added scripts to build matplotlib with numscons: >> >> http://github.com/cournape/matplotlib/tree/scons_build >> >> > Hi David, > > I'm trying to download your git repository, but I'm having trouble > cloning it. > > Can you do this? "git clone git://github.com/cournape/matplotlib.git" > Yes, I have exactly the same problem - cloning through ssh is ok, though (although it does not help you much). I already tried to remove/refork the repo from you (I thought the problem may have been caused by astraw/matplotlib being a new repo), to no end. I wish git would improve their network support... > For me, I get an error: > > $ git clone git://github.com/cournape/matplotlib.git > Initialized empty Git repository in /home/astraw/tmp/mpl/matplotlib/.git/ > fatal: The remote end hung up unexpectedly > > This has been happening for 2 days now. > > If it's not working for you, can you file a support ticket with github? > (Or just delete your old repo and start a new one on github, preferably > forking from the astraw/matplotlib.git repo.) > I will, David -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [PATCH] experimental numscons support in matplotlib
Andrew Straw wrote:
> David Cournapeau wrote:
>
>> Hi,
>>
>> I don't know if that's of any interest for matplotlib developers,
>> but I added scripts to build matplotlib with numscons:
>>
>> http://github.com/cournape/matplotlib/tree/scons_build
>>
>>
> OK, I managed to clone your repo -- I cloned mine, then added yours as a
> remote, fetched it, and pushed the results to a new branch on my github
> repo: http://github.com/astraw/matplotlib/tree/dev/cournapeau-scons-build
>
Ok - I still don't understand why it does not work the usual way, though.
> But having done that, now I'm having trouble building. Calling with
> "python setupscons.py install", I get:
>
> Traceback (most recent call last):
> File "setupscons.py", line 232, in
> setup_package()
> File "setupscons.py", line 228, in setup_package
> configuration=configuration)
> File "/usr/lib/python2.6/dist-packages/numpy/distutils/core.py", line
> 150, in setup
> config = configuration()
> File "setupscons.py", line 197, in configuration
> config.add_sconscript('SConstruct', package_path='lib/matplotlib')
> TypeError: add_sconscript() got an unexpected keyword argument
> 'package_path'
>
> What version of numpy do I need for this?
You need a very recent one (a few days old): the package_path was added
for matplotlib, as it uses a slightly unusual source. organization,
cheers,
David
--
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [PATCH] experimental numscons support in matplotlib
Andrew Straw wrote: > Michael Droettboom wrote: >> I know it's been a while since you announced this, but I'm just >> looking into this now. > > Also, I got some ways in making the buildbot build with numscons, but > I stopped at a bug where it looked like the matplotlib.tests.* modules > were not getting installed: > > http://mpl-buildbot.code.astraw.com/builders/Ubuntu%208.04%2C%20Python%202.5%2C%20amd64%2C%20scons/builds/13/steps/test/logs/stdio > > I will look at it. I would like to get some kind of automated testing for matplotlib on windows 64 (which is built using the numscons build), so I have the incentive :) > I haven't had a chance to debug this further, but I'm open to ideas. > Also, this branch is building from a git repository (a mirror of > David's which I can't clone normally, for some reason), for what it's > worth. I don't know why I have those problems either. Do you think it would be possible to just apply the patch suite to trunk in svn once we fix the test issue ? Since the patches do not touch the existing source tree (except for a few bugs on windows I can split up if required), it would be more practical to have all this in svn. cheers, David -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [PATCH] experimental numscons support in matplotlib
Andrew Straw wrote: > I looked a little further, and it depends on the directory that the > tests are run from -- if I manually log into the build slave, I can > get the tests to run (in fact, one segfaults) if I try from a > different working directory. Anyhow, now that I have a handle on it, I > think I can probably get it working... Give me a couple days. great. > As far as I'm concerned, that would be fine. > > Is PyMODINIT_FUNC pulled in from Python.h? Yes - the init func should always be 'tagged' by this macro. On Unix, any function is exported in a shared library by default, but the behavior is the opposite on windows, where a function without it will not be visible from external code. David -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
[matplotlib-devel] Announcing toydist, improving distribution and packaging situation
(warning, long post) Hi there, As some of you already know, the packaging and distributions of scientific python packages have been a constant source of frustration. Open source is about making it easy for anyone to use software how they see fit, and I think python packaging infrastructure has not been very successfull for people not intimately familiar with python. A few weeks ago, after Guido visited Berkeley and was told how those issues were still there for the scientific community, he wrote an email asking whether current efforts on distutils-sig will be enough (see http://aspn.activestate.com/ASPN/Mail/Message/distutils-sig/3775972). Several of us have been participating to this discussion, but I feel like the divide between current efforts on distutils-sig and us (the SciPy community) is not getting smaller. At best, their efforts will be more work for us to track the new distribute fork, and more likely, it will be all for nothing as it won't solve any deep issue. To be honest, most of what is considered on distutils-sig sounds like anti-goals to me. Instead of keeping up with the frustrating process of "improving" distutils, I think we have enough smart people and manpower in the scientific community to go with our own solution. I am convinced it is doable because R or haskell, with a much smaller community than python, managed to pull out something with is miles ahead compared to pypi. The SciPy community is hopefully big enough so that a SciPy-specific solution may reach critical mass. Ideally, I wish we had something with the following capabilities: - easy to understand tools - http-based package repository ala CRAN, which would be easy to mirror and backup (through rsync-like tools) - decoupling the building, packaging and distribution of code and data - reliable install/uninstall/query of what is installed locally - facilities for building windows/max os x binaries - making the life of OS vendors (Linux, *BSD, etc...) easier The packaging part == Speaking is easy, so I started coding part of this toolset, called toydist (temporary name), which I presented at Scipy India a few days ago: http://github.com/cournape/toydist/ Toydist is more or less a rip off of cabal (http://www.haskell.org/cabal/), and consist of three parts: - a core which builds a package description from a declarative file similar to cabal files. The file is almost purely declarative, and can be parsed so that no arbitrary code is executed, thus making it easy to sandbox packages builds (e.g. on a build farm). - a set of command line tools to configure, build, install, build installers (egg only for now) etc... from the declarative file - backward compatibility tools: a tool to convert existing setup.py to the new format has been written, and a tool to use distutils through the new format for backward compatibility with complex distutils extensions should be relatively easy. The core idea is to make the format just rich enough to describe most packages out there, but simple enough so interfacing it with external tools is possible and reliable. As a regular contributor to scons, I am all too aware that a build tool is a very complex beast to get right, and repeating their efforts does not make sense. Typically, I envision that complex packages such as numpy, scipy or matplotlib would use make/waf/scons for the build - in a sense, toydist is written so that writing something like numscons would be easier. OTOH, most if not all scikits should be buildable from a purely declarative file. To give you a feel of the format, here is a snippet for the grin package from Robert K. (automatically converted): Name: grin Version: 1.1.1 Summary: A grep program configured the way I like it. Description: grin I wrote grin to help me search directories full of source code. The venerable GNU grep_ and find_ are great tools, but they fall just a little short for my normal use cases. License: BSD Platforms: UNKNOWN Classifiers: License :: OSI Approved :: BSD License, Development Status :: 5 - Production/Stable, Environment :: Console, Intended Audience :: Developers, Operating System :: OS Independent, Programming Language :: Python, Topic :: Utilities, ExtraSourceFiles: README.txt, setup.cfg, setup.py, Library: InstallDepends: argparse, Modules: grin, Executable: grin module: grin function: grin_main Executable: grind module: grin function: grind_main Although still very much experimental at this point, toydist already makes some things much easier than with distutils/setuptools: - path customization for any target can be done easily: you can easily add an option in the file so that configure --mynewdir=value works and is accessible at every step. - making packages FHS compliant is not a PITA anymore, and the scheme can be adapted to any OS, be it traditional FHS-like unix,
Re: [matplotlib-devel] Announcing toydist, improving distribution and packaging situation
On Mon, Dec 28, 2009 at 11:47 PM, Stefan Schwarzburg wrote: > Hi, > I would like to add a comment from the user perspective: > > - the main reason why I'm not satisfied with pypi/distutils/etc. and why I > will not be satisfied with toydist (with the features you listed), is that > they break my installation (debian/ubuntu). Toydist (or distutils) does not break anything as is. It would be like saying make breaks debian - it does not make much sense. As stated, one of the goal of giving up distutils is to make packaging by os vendors easier. In particular, by allowing to follow the FHS, and making things more consistent. It should be possible to automatically convert most packages to .deb (or .rpm) relatively easily. When you look at the numpy .deb package, most of the issues are distutils issues, and almost everything else can be done automatically. Note that even ignoring the windows problem, there are systems to do the kind of things I am talking about for linux-only systems (the opensuse build service), because distributions are not always really good at tracking fast changing softwares. IOW, traditional linux packaging has some issues as well. And anyway, nothing prevents debian or other OS vendors to package things as they want (as they do for R packages). David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-dev] Announcing toydist, improving distribution and packaging situation
On Tue, Dec 29, 2009 at 3:03 AM, Neal Becker wrote: > David Cournapeau wrote: > >> On Mon, Dec 28, 2009 at 11:47 PM, Stefan Schwarzburg >> wrote: >>> Hi, >>> I would like to add a comment from the user perspective: >>> >>> - the main reason why I'm not satisfied with pypi/distutils/etc. and why >>> I will not be satisfied with toydist (with the features you listed), is >>> that they break my installation (debian/ubuntu). >> >> Toydist (or distutils) does not break anything as is. It would be like >> saying make breaks debian - it does not make much sense. As stated, >> one of the goal of giving up distutils is to make packaging by os >> vendors easier. In particular, by allowing to follow the FHS, and >> making things more consistent. It should be possible to automatically >> convert most packages to .deb (or .rpm) relatively easily. When you >> look at the numpy .deb package, most of the issues are distutils >> issues, and almost everything else can be done automatically. >> >> Note that even ignoring the windows problem, there are systems to do >> the kind of things I am talking about for linux-only systems (the >> opensuse build service), because distributions are not always really >> good at tracking fast changing softwares. IOW, traditional linux >> packaging has some issues as well. And anyway, nothing prevents debian >> or other OS vendors to package things as they want (as they do for R >> packages). >> >> David > > I think the breakage that is referred to I can describe on my favorite > system, fedora. > > I can install the fedora numpy rpm using yum. I could also use > easy_install. Unfortunately: > 1) Each one knows nothing about the other > 2) They may install things into conflicting paths. In particular, on fedora > arch-dependent things go in /usr/lib64/python/site-packages while > arch-independent goes into /usr/lib/python... If you mix yum with > easy_install (or setuptools), you many times wind up with 2 versions and a > lot of confusion. > > This is NOT unusual. Let's say I have numpy-1.3.0 installed from rpms. I > see the announcement of numpy-1.4.0, and decide I want it, before the rpm is > available, so I use easy_install. Now numpy-1.4.0 shows up as a standard > rpm, and a subsequent update (which could be automatic!) could produce a > broken system. Several points: - First, this is caused by distutils misfeature of defaulting to /usr. This is a mistake. It should default to /usr/local, as does every other install method from sources. - A lot of instructions start by sudo easy_install... This is a very bad advice, especially given the previous issue. > I don't really know what could be done about it. Perhaps a design that > attempts to use native backends for installation where available? The idea would be that for a few major distributions at least, you would have .rpm available on the repository. If you install from sources, there would be a few mechanisms to avoid your exact issue (like maybe defaulting to --user kind of installs). Of course, it can only be dealt up to a point. David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Tue, Dec 29, 2009 at 3:49 AM, Dag Sverre Seljebotn wrote: > > Do you here mean automatic generation of Ubuntu debs, Debian debs, Windows > MSI installer, Windows EXE installer, and so on? (If so then great!) Yes (although this is not yet implemented). In particular on windows, I want to implement a scheme so that you can convert from eggs to .exe and vice et versa, so people can still install as exe (or msi), even though the method would default to eggs. > If this is the goal, I wonder if one looks outside of Python-land one > might find something that already does this -- there's a lot of different > package format, "Linux meta-distributions", "install everywhere packages" > and so on. Yes, there are things like 0install or autopackage. I think those are deemed to fail, as long as it is not supported thoroughly by the distribution. Instead, my goal here is much simpler: producing rpm/deb. It does not solve every issue (install by non root, multiple // versions), but one has to be realistic :) I think automatically built rpm/deb, easy integration with native method can solve a lot of issues already. > > - Currently I'm making a Sage SPKG for CHOLMOD. This essentially gets the > job done by not bothering about the problem, not even using the > OS-installed Python. > > Something that would spit out both Sage SPKGs, Ubuntu debs, Windows > installers, both with Python code and C/Fortran code or a mix (and put > both in the place preferred by the system in question), seems ideal. Of > course one would still need to make sure that the code builds properly > everywhere, but just solving the distribution part of this would be a huge > step ahead. On windows, this issue may be solved using eggs: enstaller has a feature where dll put in a special location of an egg are installed in python such as they are found by the OS loader. One could have mechanisms based on $ORIGIN + rpath on linux to solve this issue for local installs on Linux, etc... But again, one has to be realistic on the goals. With toydist, I want to remove all the pile of magic, hacks built on top of distutils so that people can again hack their own solutions, as it should have been from the start (that's a big plus of python in general). It won't magically solve every issue out there, but it would hopefully help people to make their own. Bundling solutions like SAGE, EPD, etc... are still the most robust ways to deal with those issues in general, and I do not intended to replace those. > What I'm saying is that this is a software distribution problem in > general, and I'm afraid that Python-specific solutions are too narrow. Distribution is a hard problem. Instead of pushing a very narrow (and mostly ill-funded) view of how people should do things like distutils/setuptools/pip/buildout do, I want people to be able to be able to build their own solutions. No more "use this magic stick v 4.0.3.3.14svn1234, trust me it work you don't have to understand" which is too prevalant with those tools, which has always felt deeply unpythonic to me. David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-dev] Announcing toydist, improving distribution and packaging situation
On Tue, Dec 29, 2009 at 8:02 AM, Gael Varoquaux wrote: > On Mon, Dec 28, 2009 at 02:29:24PM -0500, Neal Becker wrote: >> Perhaps this could be useful: >> http://checkinstall.izto.org/ > > Yes, checkinstall is really cool. However, I tend to prefer things with > no magic that I don't have to sandbox to know what they are doing. I am still not sure the design is entirely right, but the install command in toymaker just reads a build manifest, which is a file containing all the files necessary for install. It is explicit, and list every file to be installed. By design, it cannot install anything outside this manifest. That's also how eggs are built (and soon win installers and mac os x pkg). cheers, David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield wrote: > Hi, > > In the toydist proposal/release notes, I would address 'what does > toydist do better' more explicitly. > > > > A big problem for science users is that numpy does not work with > pypi + (easy_install, buildout or pip) and python 2.6. > > > > Working with the rest of the python community as much as possible is > likely a good goal. Yes, but it is hopeless. Most of what is being discussed on distutils-sig is useless for us, and what matters is ignored at best. I think most people on distutils-sig are misguided, and I don't think the community is representative of people concerned with packaging anyway - most of the participants seem to be around web development, and are mostly dismissive of other's concerns (OS packagers, etc...). I want to note that I am not starting this out of thin air - I know most of distutils code very well, I have been the mostly sole maintainer of numpy.distutils for 2 years now. I have written extensive distutils extensions, in particular numscons which is able to fully build numpy, scipy and matplotlib on every platform that matters. Simply put, distutils code is horrible (this is an objective fact) and flawed beyond repair (this is more controversial). IMHO, it has almost no useful feature, except being standard. If you want a more detailed explanation of why I think distutils and all tools on top are deeply flawed, you can look here: http://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/ > numpy used to work with buildout in python2.5, but not with 2.6. > buildout lets other team members get up to speed with a project by > running one command. It installs things in the local directory, not > system wide. So you can have different dependencies per project. I don't think it is a very useful feature, honestly. It seems to me that they created a huge infrastructure to split packages into tiny pieces, and then try to get them back together, imaganing that multiple installed versions is a replacement for backward compatibility. Anyone with extensive packaging experience knows that's a deeply flawed model in general. > Plenty of good work is going on with python packaging. That's the opposite of my experience. What I care about is: - tools which are hackable and easily extensible - robust install/uninstall - real, DAG-based build system - explicit and repeatability None of this is supported by the tools, and the current directions go even further away. When I have to explain at length why the command-based design of distutils is a nightmare to work with, I don't feel very confident that the current maintainers are aware of the issues, for example. It shows that they never had to extend distutils much. > > There are build farms for windows packages and OSX uploaded to pypi. > Start uploading pre releases to pypi, and you get these for free (once > you make numpy compile out of the box on those compile farms). There > are compile farms for other OSes too... like ubuntu/debian, macports > etc. Some distributions even automatically download, compile and > package new releases once they spot a new file on your ftp/web site. I am familiar with some of those systems (PPA and opensuse build service in particular). One of the goal of my proposal is to make it easier to interoperate with those tools. I think Pypi is mostly useless. The lack of enforced metadata is a big no-no IMHO. The fact that Pypi is miles beyond CRAN for example is quite significant. I want CRAN for scientific python, and I don't see Pypi becoming it in the near future. The point of having our own Pypi-like server is that we could do the following: - enforcing metadata - making it easy to extend the service to support our needs > > pypm: http://pypm.activestate.com/list-n.html#numpy It is interesting to note that one of the maintainer of pypm has recently quitted the discussion about Pypi, most likely out of frustration from the other participants. > Documentation projects are being worked on to document, give tutorials > and make python packaging be easier all round. As witnessed by 20 or > so releases on pypi every day(and growing), lots of people are using > the python packaging tools successfully. This does not mean much IMO. Uploading on Pypi is almost required to use virtualenv, buildout, etc.. An interesting metric is not how many packages are uploaded, but how much it is used outside developers. > > I'm not sure making a separate build tool is a good idea. I think > going with the rest of the python community, and improving the tools > there is a better idea. It has been tried, and IMHO has been proved to have failed. You can look at the recent discussion (the one started by Guido in particular). > pps. some notes on toydist itself. > - toydist convert is cool for people converting a setup.py . This > means that most people can try out toydist right away. but
Re: [matplotlib-devel] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield wrote: > Buildout is what a lot of the python community are using now. I would like to note that buildout is a solution to a problem that I don't care to solve. This issue is particularly difficult to explain to people accustomed with buildout in my experience - I have not found a way to explain it very well yet. Buildout, virtualenv all work by sandboxing from the system python: each of them do not see each other, which may be useful for development, but as a deployment solution to the casual user who may not be familiar with python, it is useless. A scientist who installs numpy, scipy, etc... to try things out want to have everything available in one python interpreter, and does not want to jump to different virtualenvs and whatnot to try different packages. This has strong consequences on how you look at things from a packaging POV: - uninstall is crucial - a package bringing down python is a big no no (this happens way too often when you install things through setuptools) - if something fails, the recovery should be trivial - the person doing the installation may not know much about python - you cannot use sandboxing as a replacement for backward compatibility (that's why I don't care much about all the discussion about versioning - I don't think it is very useful as long as python itself does not support it natively). In the context of ruby, this article makes a similar point: http://www.madstop.com/ruby/ruby_has_a_distribution_problem.html David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Wed, Dec 30, 2009 at 3:36 AM, René Dudfield wrote: > On Tue, Dec 29, 2009 at 2:34 PM, David Cournapeau wrote: >> On Tue, Dec 29, 2009 at 10:27 PM, René Dudfield wrote: >> >>> Buildout is what a lot of the python community are using now. >> >> I would like to note that buildout is a solution to a problem that I >> don't care to solve. This issue is particularly difficult to explain >> to people accustomed with buildout in my experience - I have not found >> a way to explain it very well yet. > > Hello, > > The main problem buildout solves is getting developers up to speed > very quickly on a project. They should be able to call one command > and get dozens of packages, and everything else needed ready to go, > completely isolated from the rest of the system. > > If a project does not want to upgrade to the latest versions of > packages, they do not have to. This reduces the dependency problem a > lot. As one package does not have to block on waiting for 20 other > packages. It makes iterating packages daily, or even hourly to not be > a problem - even with dozens of different packages used. This is not > theoretical, many projects iterate this quickly, and do not have > problems. > > Backwards compatibility is of course a great thing to keep up... but > harder to do with dozens of packages, some of which are third party > ones. For example, some people are running pygame applications > written 8 years ago that are still running today on the latest > versions of pygame. I don't think people in the python world > understand API, and ABI compatibility as much as those in the C world. > > However buildout is a solution to their problem, and allows them to > iterate quickly with many participants, on many different projects. > Many of these people work on maybe 20-100 different projects at once, > and some machines may be running that many applications at once too. > So using the system pythons packages is completely out of the question > for them. This is all great, but I don't care about solving this issue, this is a *developer* issue. I don't mean this is not an important issue, it is just totally out of scope. The developer issues I care about are much more fine-grained (corrent dependency handling between target, toolchain customization, etc...). Note however that hopefully, by simplifying the packaging tools, the problems you see with numpy on 2.6 would be less common. The whole distutils/setuptools/distribute stack is hopelessly intractable, given how messy the code is. > > It is very easy to include a dozen packages in a buildout, so that you > have all the packages required. I think there is a confusion - I mostly care about *end users*. People who may not have compilers, who want to be able to easily upgrade one package, etc... David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Wed, Dec 30, 2009 at 8:15 PM, René Dudfield wrote:
>
> Sitting down with Tarek(who is one of the current distutils
> maintainers) in Berlin we had a little discussion about packaging over
> pizza and beer... and he was quite mindful of OS packagers problems
> and issues.
This has been said many times on distutils-sig, but no concrete action
has ever been taken in that direction. For example, toydist already
supports the FHS better than distutils, and is more flexible. I have
tried several times to explain why this matters on distutils-sig, but
you then have the peanuts gallery interfering with unrelated nonsense
(like it would break windows, as if it could not be implemented
independently).
Also, retrofitting support for --*dir in distutils would be *very*
difficult, unless you are ready to break backward compatibility (there
are 6 ways to install data files, and each of them has some corner
cases, for example - it is a real pain to support this correctly in
the convert command of toydist, and you simply cannot recover missing
information to comply with the FHS in every case).
>
> However these systems were developed by the zope/plone/web crowd, so
> they are naturally going to be thinking a lot about zope/plone/web
> issues.
Agreed - it is natural that they care about their problems first,
that's how it works in open source. What I find difficult is when our
concern are constantly dismissed by people who have no clue about our
issues - and later claim we are not cooperative.
> Debian, and ubuntu packages for them are mostly useless
> because of the age.
That's where the build farm enters. This is known issue, that's why
the build service or PPA exist in the first place.
> I think
> perhaps if toydist included something like stdeb as not an extension
> to distutils, but a standalone tool (like toydist) there would be less
> problems with it.
That's pretty much how I intend to do things. Currently, in toydist,
you can do something like:
from toydist.core import PackageDescription
pkg = PackageDescription.from_file("toysetup.info")
# pkg now gives you access to metadata, as well as extensions, python
modules, etc...
I think this gives almost everything that is needed to implement a
sdist_dsc command. Contrary to the Distribution class in distutils,
this class would not need to be subclassed/monkey-patched by
extensions, as it only cares about the description, and is 100 %
uncoupled from the build part.
> yes, I have also battled with distutils over the years. However it is
> simpler than autotools (for me... maybe distutils has perverted my
> fragile mind), and works on more platforms for python than any other
> current system.
Autotools certainly works on more platforms (windows notwhistanding),
if only because python itself is built with autoconf. Distutils
simplicity is a trap: it is simpler only if you restrict to what
distutils gives you. Don't get me wrong, autotools are horrible, but I
have never encountered cases where I had to spend hours to do trivial
tasks, as has been the case with distutils. Numpy build system would
be much, much easier to implement through autotools, and would be much
more reliable.
> However
> distutils has had more tests and testing systems added, so that
> refactoring/cleaning up of distutils can happen more so.
You can't refactor distutils without breaking backward compatibility,
because distutils has no API. The whole implementation is the API.
That's one of the fundamental disagreement I and other scipy dev have
with current contributors on distutils-sig: the starting point
(distutils) and the goal are so far away from each other that getting
there step by step is hopeless.
> I agree with many things in that post. Except your conclusion on
> multiple versions of packages in isolation. Package isolation is like
> processes, and package sharing is like threads - and threads are evil!
I don't find the comparison very helpful (for once, you can share data
between processes, whereas virtualenv cannot see each other AFAIK).
> Science is supposed to allow repeatability. Without the same versions
> of packages, repeating experiments is harder. This is a big problem
> in science that multiple versions of packages in _isolation_ can help
> get to a solution to the repeatability problem.
I don't think that's true - at least it does not reflect my experience
at all. But then, I don't pretend to have an extensive experience
either. From most of my discussions at scipy conferences, I know most
people are dissatisfied with the current python solutions.
>
>>> Plenty of good work is going on with python packaging.
>>
>> That's the opposite of my experience. What I care about is:
>> - tools which are hackable and easily extensible
>> - robust install/uninstall
>> - real, DAG-based build system
>> - explicit and repeatability
>>
>> None of this is supported by the tools, and the current directions go
>> even further away. When I have to explain at length w
Re: [matplotlib-devel] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Wed, Dec 30, 2009 at 11:26 PM, Darren Dale wrote: > Hi David, > > On Mon, Dec 28, 2009 at 9:03 AM, David Cournapeau wrote: >> Executable: grin >> module: grin >> function: grin_main >> >> Executable: grind >> module: grin >> function: grind_main > > Have you thought at all about operations that are currently performed > by post-installation scripts? For example, it might be desirable for > the ipython or MayaVi windows installers to create a folder in the > Start menu that contains links the the executable and the > documentation. This is probably a secondary issue at this point in > toydist's development, but I think it is an important feature in the > long run. The main problem I see with post hooks is how to support them in installers. For example, you would have a function which does the post install, and declare it as a post install hook through decorator: @hook.post_install def myfunc(): pass The main issue is how to communicate data - that's a major issue in every build system I know of (scons' solution is ugly: every function takes an env argument, which is basically a giant global variable). > > Also, have you considered support for package extras (package variants > in Ports, allowing you to specify features that pull in additional > dependencies like traits[qt4])? Enthought makes good use of them in > ETS, and I think they would be worth keeping. The declarative format may declare flags as follows: Flag: c_exts Description: Build (optional) C extensions Default: false Library: if flag(c_exts): Extension: foo sources: foo.c And this is automatically available at configure stage. It can be used anywhere in Library, not just for Extension (you could use is within the Requires section). I am considering adding more than Flag (flag are boolean), if it does not make the format too complex. The use case I have in mind is something like: toydist configure --with-lapack-dir=/opt/mkl/lib which I have wished to implement for numpy for ages. David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Wed, Dec 30, 2009 at 11:26 PM, Darren Dale wrote: > Hi David, > > On Mon, Dec 28, 2009 at 9:03 AM, David Cournapeau wrote: >> Executable: grin >> module: grin >> function: grin_main >> >> Executable: grind >> module: grin >> function: grind_main > > Have you thought at all about operations that are currently performed > by post-installation scripts? For example, it might be desirable for > the ipython or MayaVi windows installers to create a folder in the > Start menu that contains links the the executable and the > documentation. This is probably a secondary issue at this point in > toydist's development, but I think it is an important feature in the > long run. > > Also, have you considered support for package extras (package variants > in Ports, allowing you to specify features that pull in additional > dependencies like traits[qt4])? Enthought makes good use of them in > ETS, and I think they would be worth keeping. Does this example covers what you have in mind ? I am not so familiar with this feature of setuptools: Name: hello Version: 1.0 Library: BuildRequires: paver, sphinx, numpy if os(windows) BuildRequires: pywin32 Packages: hello Extension: hello._bar sources: src/hellomodule.c if os(linux) Extension: hello._linux_backend sources: src/linbackend.c Note that instead of os(os_name), you can use flag(flag_name), where flag are boolean variables which can be user defined: http://github.com/cournape/toydist/blob/master/examples/simples/conditional/toysetup.info http://github.com/cournape/toydist/blob/master/examples/var_example/toysetup.info David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Thu, Dec 31, 2009 at 6:06 AM, Darren Dale wrote: > > I should defer to the description of extras in the setuptools > documentation. It is only a few paragraphs long: > > http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies Ok, so there are two issues related to this feature: - supporting variant at the build stage - supporting different variants of the same package in the dependency graph at install time The first issue is definitely supported - I fixed a bug in toydist to support this correctly, and this will be used when converting setuptools-based setup.py which use the features argument. The second issue is more challenging. It complicates the dependency handling quite a bit, and may cause difficult situations to happen at dependency resolution time. This becomes particularly messy if you mix packages you build yourself with packages grabbed from a repository. I wonder if there is a simpler solution which would give a similar feature set. cheers, David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-User] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Fri, Jan 1, 2010 at 10:43 PM, Pierre Raybaut wrote: > Hi David, > > Following your announcement for the 'toydist' module, I think that > your project is very promising: this is certainly a great idea and it > will be very controversial but that's because people expectactions are > great on this matter (distutils is so disappointing indeed). > > Anyway, if I may be useful, I'll gladly contribute to it. > In time, I could change the whole Python(x,y) packaging system (which > is currently quite ugly... but easy/quick to manage/maintain) to > use/promote this new module. That would be a good way to test toydist on a real, complex package. I am not familiar at all with python(x,y) internals. Do you have some explanation I could look at somewhere ? In the meantime, I will try to clean-up the code to have a first experimental release. cheers, David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Sat, Jan 2, 2010 at 4:58 PM, Gael Varoquaux wrote: > On Sat, Jan 02, 2010 at 11:32:00AM +0900, David Cournapeau wrote: >> [snip] >> - supporting different variants of the same package in the >> dependency graph at install time > >> [snip] > >> The second issue is more challenging. It complicates the dependency >> handling quite a bit, and may cause difficult situations to happen at >> dependency resolution time. This becomes particularly messy if you mix >> packages you build yourself with packages grabbed from a repository. I >> wonder if there is a simpler solution which would give a similar >> feature set. > > > AFAICT, in Debian, the same feature is given via virtual packages: you > would have: I don't think virtual-packages entirely fix the issue. AFAIK, virtual packages have two uses: - handle dependencies where several packages may resolve one particular dependency in an equivalent way (one good example is LAPACK: both liblapack and libatlas provides the lapack feature) - closer to this discussion, you can build several variants of the same package, and each variant would resolve the dependency on a virtual package handling the commonalities. For example, say we have two numpy packages, one built with lapack (python-numpy-full), the other without (python-numpy-core). What happens when a package foo depends on numpy-full, but numpy-core is installed ? AFAICS, this can only work as long as the set containing every variant can be ordered (in the conventional set ordering sense), and the dependency can be satisfied by the smallest one. cheers, David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [SciPy-dev] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Sun, Jan 3, 2010 at 3:27 AM, Andrew Straw wrote: >> > Typically, the dependencies only depend on the smallest subset of what > they require (if they don't need lapack, they'd only depend on > python-numpy-core in your example), but yes, if there's an unsatisfiable > condition, then apt-get will raise an error and abort. In practice, this > system seems to work quite well, IMO. Yes, but: - debian dependency resolution is complex. I think many people don't realize how complex the problem really is (AFAIK, any correct scheme to resolve dependencies in debian requires an algorithm which is NP-complete ) - introducing a lot of variants significantly slow down the whole thing. I think it worths thinking whether our problems warrant such a complexity. > > Anyhow, here's the full Debian documentation: > http://www.debian.org/doc/debian-policy/ch-relationships.html This is not the part I am afraid of. This is: http://people.debian.org/~dburrows/model.pdf cheers, David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Sun, Jan 3, 2010 at 8:05 PM, Nathaniel Smith wrote: > On Tue, Dec 29, 2009 at 6:34 AM, David Cournapeau wrote: >> Buildout, virtualenv all work by sandboxing from the system python: >> each of them do not see each other, which may be useful for >> development, but as a deployment solution to the casual user who may >> not be familiar with python, it is useless. A scientist who installs >> numpy, scipy, etc... to try things out want to have everything >> available in one python interpreter, and does not want to jump to >> different virtualenvs and whatnot to try different packages. > > What I do -- and documented for people in my lab to do -- is set up > one virtualenv in my user account, and use it as my default python. (I > 'activate' it from my login scripts.) The advantage of this is that > easy_install (or pip) just works, without any hassle about permissions > etc. It just works if you happen to be able to build everything from sources. That alone means you ignore the majority of users I intend to target. No other community (except maybe Ruby) push those isolated install solutions as a general deployment solutions. If it were such a great idea, other people would have picked up those solutions. > This should be easier, but I think the basic approach is sound. > "Integration with the package system" is useless; the advantage of > distribution packages is that distributions can provide a single > coherent system with consistent version numbers across all packages, > etc., and the only way to "integrate" with that is to, well, get the > packages into the distribution. Another way is to provide our own repository for a few major distributions, with automatically built packages. This is how most open source providers work. Miguel de Icaza explains this well: http://tirania.org/blog/archive/2007/Jan-26.html I hope we will be able to reuse much of the opensuse build service infrastructure. > > On another note, I hope toydist will provide a "source prepare" step, > that allows arbitrary code to be run on the source tree. (For, e.g., > cython->C conversion, ad-hoc template languages, etc.) IME this is a > very common pain point with distutils; there is just no good way to do > it, and it has to be supported in the distribution utility in order to > get everything right. In particular: > -- Generated files should never be written to the source tree > itself, but only the build directory > -- Building from a source checkout should run the "source prepare" > step automatically > -- Building a source distribution should also run the "source > prepare" step, and stash the results in such a way that when later > building the source distribution, this step can be skipped. This is a > common requirement for user convenience, and necessary if you want to > avoid arbitrary code execution during builds. Build directories are hard to implement right. I don't think toydist will support this directly. IMO, those advanced builds warrant a real build tool - one main goal of toydist is to make integration with waf or scons much easier. Both waf and scons have the concept of a build directory, which should do everything you described. David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Mon, Jan 4, 2010 at 8:42 AM, Nathaniel Smith wrote: > On Sun, Jan 3, 2010 at 4:23 AM, David Cournapeau wrote: >> On Sun, Jan 3, 2010 at 8:05 PM, Nathaniel Smith wrote: >>> What I do -- and documented for people in my lab to do -- is set up >>> one virtualenv in my user account, and use it as my default python. (I >>> 'activate' it from my login scripts.) The advantage of this is that >>> easy_install (or pip) just works, without any hassle about permissions >>> etc. >> >> It just works if you happen to be able to build everything from >> sources. That alone means you ignore the majority of users I intend to >> target. >> >> No other community (except maybe Ruby) push those isolated install >> solutions as a general deployment solutions. If it were such a great >> idea, other people would have picked up those solutions. > > AFAICT, R works more-or-less identically (once I convinced it to use a > per-user library directory); install.packages() builds from source, > and doesn't automatically pull in and build random C library > dependencies. As mentioned by Robert, this is different from the usual virtualenv approach. Per-user app installation is certainly a useful (and uncontroversial) feature. And R does support automatically-built binary installers. > > Sure, I'm aware of the opensuse build service, have built third-party > packages for my projects, etc. It's a good attempt, but also has a lot > of problems, and when talking about scientific software it's totally > useless to me :-). First, I don't have root on our compute cluster. True, non-root install is a problem. Nothing *prevents* dpkg to run in non root environment in principle if the packages itself does not require it, but it is not really supported by the tools ATM. > Second, even if I did I'd be very leery about installing third-party > packages because there is no guarantee that the version numbering will > be consistent between the third-party repo and the real distro repo -- > suppose that the distro packages 0.1, then the third party packages > 0.2, then the distro packages 0.3, will upgrades be seamless? What if > the third party screws up the version numbering at some point? Debian > has "epochs" to deal with this, but third-parties can't use them and > maintain compatibility. Actually, at least with .deb-based distributions, this issue has a solution. As packages has their own version in addition to the upstream version, PPA-built packages have their own versions. https://help.launchpad.net/Packaging/PPA/BuildingASourcePackage Of course, this assumes a simple versioning scheme in the first place, instead of the cluster-fck that versioning has became within python packages (again, the scheme used in python is much more complicated than everyone else, and it seems that nobody has ever stopped and thought 5 minutes about the consequences, and whether this complexity was a good idea in the first place). > What if the person making the third party > packages is not an expert on these random distros that they don't even > use? I think simple rules/conventions + build farms would solve most issues. The problem is if you allow total flexibility as input, then automatic and simple solutions become impossible. Certainly, PPA and the build service provides for a much better experience than anything pypi has ever given to me. > Third, while we shouldn't advocate that people screw up backwards > compatibility, version skew is a real issue. If I need one version of > a package and my lab-mate needs another and we have submissions due > tomorrow, then filing bugs is a great idea but not a solution. Nothing prevents you from using virtualenv in that case (I may sound dismissive of those tools, but I am really not. I use them myselves. What I strongly react to is when those are pushed as the de-facto, standard method). > Fourth, > even if we had expert maintainers taking care of all these third-party > packages and all my concerns were answered, I couldn't convince our > sysadmin of that; he's the one who'd have to clean up if something > went wrong we don't have a big budget for overtime. I am not advocating using only packaged, binary installers. I am advocating using them as much as possible where it makes sense - on windows and mac os x in particular. Toydist also aims at making it easier to build, customize installs. Although not yet implemented, --user-like scheme would be quite simple to implement, because toydist installer internally uses autoconf-like directories description (of which --user is a special case). If you need sandboxed installs, customized installs, toydist will not prevent it. It is certainly my intention to make it possible to use vir
Re: [matplotlib-devel] [Numpy-discussion] Announcing toydist, improving distribution and packaging situation
On Mon, Jan 4, 2010 at 5:48 PM, Dag Sverre Seljebotn wrote: > > Rolling this into the Python package distribution scheme seems backwards > though, since a lot of binary packages that have nothing to do with Python > are used as well Yep, exactly. > > To solve the exact problem you (and me) have I think the best solution is > to integrate the tools mentioned above with what David is planning (SciPI > etc.). Or if that isn't good enough, find generic "userland package > manager" that has nothing to do with Python (I'm sure a dozen > half-finished ones must have been written but didn't look), finish it, and > connect it to SciPI. You have 0install, autopackage and klik, to cite the ones I know about. I wish people had looked at those before rolling toy solutions to complex problems. > > What David does (I think) is seperate the concerns. Exactly - you've describe this better than I did David -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
