[Numpy-discussion] preferred numpy build system
Hi,
I have couple beginners questions about numscons. What is the
preferred build system for numpy now, is it numscons? The README
doesn't mention numscons, so I am a bit confused what the future plan
is.
Also by doing:
$ python setupscons.py install
Running from numpy source directory.
Traceback (most recent call last):
File "setupscons.py", line 56, in
raise DistutilsError('\n'.join(msg))
distutils.errors.DistutilsError: You cannot build numpy with scons
without the numscons package
(Failure was: No module named numscons)
so the numscons package needs to be installed -- but it is not in
Debian. So is it supposed to be in Debian? Is numscons supposed to be
a build system for other projects as well? Why not to just send the
needed patches to scons and just use scons?
Thanks,
Ondrej
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
Hi Ondrej,
On Sat, Feb 7, 2009 at 3:04 PM, Ondrej Certik wrote:
> Hi,
>
> I have couple beginners questions about numscons. What is the
> preferred build system for numpy now, is it numscons? The README
> doesn't mention numscons, so I am a bit confused what the future plan
> is.
>
Sorry for the confusion: numscons is NOT the preferred build system.
The current numpy.distutils extensions, as shipped by numpy, is the
preferred one. Numscons is more an experiment, if you want.
> Also by doing:
>
> $ python setupscons.py install
> Running from numpy source directory.
> Traceback (most recent call last):
> File "setupscons.py", line 56, in
>raise DistutilsError('\n'.join(msg))
> distutils.errors.DistutilsError: You cannot build numpy with scons
> without the numscons package
> (Failure was: No module named numscons)
>
>
> so the numscons package needs to be installed -- but it is not in
> Debian.
No, it not.
> So is it supposed to be in Debian?
No, I don't think it should be. It is not yet stabilized code wise, so
it does not make much sense to package it.
> Is numscons supposed to be
> a build system for other projects as well? Why not to just send the
> needed patches to scons and just use scons?
Because you cannot just use scons. Numscons is a library build on top
of scons, for the needs of numpy. There also needs to be some hook
from numpy.distutils to use scons (numscons adds a new distutils
command, which is used instead of build to build any compiled
code-based extensions). Most of the changes needed for scons have been
integrated upstream, though, except one or two things.
David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
Hi David, > Sorry for the confusion: numscons is NOT the preferred build system. > The current numpy.distutils extensions, as shipped by numpy, is the > preferred one. Numscons is more an experiment, if you want. Ah, I see, thanks for the clarification. >> So is it supposed to be in Debian? > > No, I don't think it should be. It is not yet stabilized code wise, so > it does not make much sense to package it. Ok. > >> Is numscons supposed to be >> a build system for other projects as well? Why not to just send the >> needed patches to scons and just use scons? > > Because you cannot just use scons. Numscons is a library build on top > of scons, for the needs of numpy. There also needs to be some hook > from numpy.distutils to use scons (numscons adds a new distutils > command, which is used instead of build to build any compiled > code-based extensions). Most of the changes needed for scons have been > integrated upstream, though, except one or two things. I see. I think it's a bit confusing that one needs to build a new build system just to build numpy, e.g. that both distutils and scons are not good enough. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
> I see. I think it's a bit confusing that one needs to build a new > build system just to build numpy, e.g. that both distutils and scons > are not good enough. I would not say that numscons is a *new* build system. Rather, I look at numscons as a glue layer that allows scons to be used within distutils for building extensions and libraries. distutils/numpy/distutils does things that scons doesn't and scons does things that distutils doesn't. But you definitely need the glue layer to use both approaches together. Cheers, Brian > Ondrej > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 3:21 AM, Ondrej Certik wrote: > Hi David, > >> Sorry for the confusion: numscons is NOT the preferred build system. >> The current numpy.distutils extensions, as shipped by numpy, is the >> preferred one. Numscons is more an experiment, if you want. > > Ah, I see, thanks for the clarification. > >>> So is it supposed to be in Debian? >> >> No, I don't think it should be. It is not yet stabilized code wise, so >> it does not make much sense to package it. > > Ok. > >> >>> Is numscons supposed to be >>> a build system for other projects as well? Why not to just send the >>> needed patches to scons and just use scons? >> >> Because you cannot just use scons. Numscons is a library build on top >> of scons, for the needs of numpy. There also needs to be some hook >> from numpy.distutils to use scons (numscons adds a new distutils >> command, which is used instead of build to build any compiled >> code-based extensions). Most of the changes needed for scons have been >> integrated upstream, though, except one or two things. > > I see. I think it's a bit confusing that one needs to build a new > build system just to build numpy, e.g. that both distutils and scons > are not good enough. I don't find it that surprising - numpy and scipy require some relatively advanced features (mixed language and cross-platform with support for many toolchains). Within the open source tools, I know only two which can handle those requirements: scons and cmake. For example, it would almost impossible to build numpy/scipy with autoconf. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 3:10 AM, David Cournapeau wrote: > On Sun, Feb 8, 2009 at 3:21 AM, Ondrej Certik wrote: >> Hi David, >> >>> Sorry for the confusion: numscons is NOT the preferred build system. >>> The current numpy.distutils extensions, as shipped by numpy, is the >>> preferred one. Numscons is more an experiment, if you want. >> >> Ah, I see, thanks for the clarification. >> So is it supposed to be in Debian? >>> >>> No, I don't think it should be. It is not yet stabilized code wise, so >>> it does not make much sense to package it. >> >> Ok. >> >>> Is numscons supposed to be a build system for other projects as well? Why not to just send the needed patches to scons and just use scons? >>> >>> Because you cannot just use scons. Numscons is a library build on top >>> of scons, for the needs of numpy. There also needs to be some hook >>> from numpy.distutils to use scons (numscons adds a new distutils >>> command, which is used instead of build to build any compiled >>> code-based extensions). Most of the changes needed for scons have been >>> integrated upstream, though, except one or two things. >> >> I see. I think it's a bit confusing that one needs to build a new >> build system just to build numpy, e.g. that both distutils and scons >> are not good enough. > > I don't find it that surprising - numpy and scipy require some > relatively advanced features (mixed language and cross-platform with > support for many toolchains). Within the open source tools, I know > only two which can handle those requirements: scons and cmake. For > example, it would almost impossible to build numpy/scipy with > autoconf. Yes, I am investigating cmake, it's pretty cool. I wrote some macros for cython etc. What I like about cmake is that it is cross platform and it just produces makefiles on linux, or visual studio files (or whatever) on windows. When I get more experience with it, I'll post here. What I don't like on cmake is that it uses it's own language, instead of python, on the other hand, so far everything seems to just work. Contrary to numscons, where it looks almost like a new python program just to build numpy. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
> I don't find it that surprising - numpy and scipy require some > relatively advanced features (mixed language and cross-platform with > support for many toolchains). Within the open source tools, I know > only two which can handle those requirements: scons and cmake. For > example, it would almost impossible to build numpy/scipy with > autoconf. I know some autotools/autoconf hackers that could probably get it to build numpy/scipy, ..., oh what, you want Windows support? ;-) Cheers, Brian > David > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
> Yes, I am investigating cmake, it's pretty cool. I wrote some macros > for cython etc. What I like about cmake is that it is cross platform > and it just produces makefiles on linux, or visual studio files (or > whatever) on windows. When I get more experience with it, I'll post > here. Yes, while I haven't played with cmake much, it does look very nice. > What I don't like on cmake is that it uses it's own language, instead > of python, on the other hand, so far everything seems to just work. I too don't like the idea of cmake having its own language. While I love to learn new languages, I like to have good reasons and I'm not sure that building software is a good reason. I have been playing with Scons and I really do like the fact that it is Python. While I haven't tried it yet, I am pretty sure that we could use IPython to interactively debug a Scons build. That would be really nice! > Contrary to numscons, where it looks almost like a new python program > just to build numpy. But hold on, it is not fair to compare cmake with numscons (this is an apples and oranges thing). You should compare cmake with *Scons* itself. The problem with comparing cmake with numscons is that cmake can't do what numscons does - i.e., plug into distutils. There is a lot of extra things that distutils does other than build extensions and cmake won't do these things out of the box. Obviously, someone could get cmake to do these things, but then you are developing a complete distutils replacement. And I think that any distutils replacement should done in Python. Cheers, Brian > Ondrej > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 12:26 PM, Brian Granger wrote: >> Yes, I am investigating cmake, it's pretty cool. I wrote some macros >> for cython etc. What I like about cmake is that it is cross platform >> and it just produces makefiles on linux, or visual studio files (or >> whatever) on windows. When I get more experience with it, I'll post >> here. > > Yes, while I haven't played with cmake much, it does look very nice. > >> What I don't like on cmake is that it uses it's own language, instead >> of python, on the other hand, so far everything seems to just work. > > I too don't like the idea of cmake having its own language. While I > love to learn new languages, I like to have good reasons and I'm not > sure that building software is a good reason. > > I have been playing with Scons and I really do like the fact that it > is Python. While I haven't tried it yet, I am pretty sure that we > could use IPython to interactively debug a Scons build. That would be > really nice! > >> Contrary to numscons, where it looks almost like a new python program >> just to build numpy. > > But hold on, it is not fair to compare cmake with numscons (this is an > apples and oranges thing). You should compare cmake with *Scons* > itself. The problem with comparing cmake with numscons is that cmake > can't do what numscons does - i.e., plug into distutils. There is a > lot of extra things that distutils does other than build extensions > and cmake won't do these things out of the box. Obviously, someone > could get cmake to do these things, but then you are developing a > complete distutils replacement. And I think that any distutils > replacement should done in Python. Yes, I agree with what you said. I still want to try cmake though, if nothing, at least to know where we want to go. Yes, I should compare cmake with scons. We can either develop python front end to cmake, or just help improve scons/numscons etc. I really like that cmake generates native makefiles and windows visual studio files etc. In any case, whatever we choose to use, I want to have experience with both scons and cmake. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Mon, Feb 9, 2009 at 4:39 AM, Ondrej Certik wrote: > > Yes, I am investigating cmake, it's pretty cool. I wrote some macros > for cython etc. What I like about cmake is that it is cross platform > and it just produces makefiles on linux, or visual studio files (or > whatever) on windows. When I get more experience with it, I'll post > here. That's exactly what I don't like about cmake - it means you can't produce accurate builds (you need to rerun cmake everytime you change the configuration or dependencies, whereas this is automatic with scons/waf). It also have (used to have) very poor documentation unless you buy the book - but it looks like this is changing. > What I don't like on cmake is that it uses it's own language, instead > of python, on the other hand, so far everything seems to just work. > Contrary to numscons, where it looks almost like a new python program > just to build numpy. Again, numscons is just a library on top of scons to support things we need in numpy, it is not really a new program - it is a separate package to avoid adding experimental code to numpy itself. Numscons is ~ 3000 LOC, of which 1000 is for the core, 1000 for blas/lapack/fortran support, and 1000 for tools which are not properly supported in scons (recent MSVC, python extensions). I think you would have almost as much work with cmake if not more - when I started numscons, cmake did not have fortran support (it now has, although I don't know how far - it does not seem to handle mixed fortran/C, for example). If you don't mind fast changing software, you should look at waf: it is extremely fast, based on python. It also handles a lot of distribution issues already (tarball generation, compiler selection, etc...) which scons does not. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Mon, Feb 9, 2009 at 5:17 AM, Brian Granger wrote: >> I don't find it that surprising - numpy and scipy require some >> relatively advanced features (mixed language and cross-platform with >> support for many toolchains). Within the open source tools, I know >> only two which can handle those requirements: scons and cmake. For >> example, it would almost impossible to build numpy/scipy with >> autoconf. > > I know some autotools/autoconf hackers that could probably get it to > build numpy/scipy, ..., oh what, you want Windows support? ;-) Yes, autotools support for MS compilers is very poor - I think it is fair to say it is unusable as of today. And one of the goal of numscons is to make the process easier to use/hack (it is a mixed success on this front - the build knowledge is much easier to read in numscons than in distutils IMHO, but it is still too arcane on some fronts, partly because distutils is way too flexible in areas it should not). David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
> That's exactly what I don't like about cmake - it means you can't > produce accurate builds (you need to rerun cmake everytime you change > the configuration or dependencies, whereas this is automatic with > scons/waf). It also have (used to have) very poor documentation unless > you buy the book - but it looks like this is changing. You can always rerun cmake if you want (and make it automatic). Imho that's not a problem. But maybe it is done in a better way with scons. > >> What I don't like on cmake is that it uses it's own language, instead >> of python, on the other hand, so far everything seems to just work. >> Contrary to numscons, where it looks almost like a new python program >> just to build numpy. > > Again, numscons is just a library on top of scons to support things we > need in numpy, it is not really a new program - it is a separate > package to avoid adding experimental code to numpy itself. Numscons is > ~ 3000 LOC, of which 1000 is for the core, 1000 for > blas/lapack/fortran support, and 1000 for tools which are not properly > supported in scons (recent MSVC, python extensions). > > I think you would have almost as much work with cmake if not more - > when I started numscons, cmake did not have fortran support (it now > has, although I don't know how far - it does not seem to handle mixed > fortran/C, for example). > > If you don't mind fast changing software, you should look at waf: it > is extremely fast, based on python. It also handles a lot of > distribution issues already (tarball generation, compiler selection, > etc...) which scons does not. Yes, waf is pretty cool, even though the last time I looked at it, it wasn't able to compile my project, which was larger than just couple files. But it seems it's development is progressing fast. As to the other things, one nice thing about cmake is that it is production ready right now, it is well tested (kde4) and it is in distributions (e.g. Debian). Everything else needs nontrivial adjustments, or is not as well tested (yet). Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
Ondrej Certik wrote: >> That's exactly what I don't like about cmake - it means you can't >> produce accurate builds (you need to rerun cmake everytime you change >> the configuration or dependencies, whereas this is automatic with >> scons/waf). It also have (used to have) very poor documentation unless >> you buy the book - but it looks like this is changing. >> > > You can always rerun cmake if you want (and make it automatic). Imho > that's not a problem. But maybe it is done in a better way with scons. > I think it is a problem - it means you have to update it explicitly when the configuration changes. In scons, the signature concept is quite powerful: not only are file dependencies handled, but also command lines, etc... For example, in numscons, if you change the blas/lapack from atlas to say MKL, only linking and eventually configuration changes are rebuilt. If you change the fortran compiler but not the C compiler, only fortran code is rebuilt. All of this is 100 % automatic. When testing software on many toolchains/platforms, this is very valuable - it is one of the main reason why Intel, Vmware use it, at least from the information I understood from the Mailing list. In terms of accuracy, scons is way beyond anything cmake has to offer. In the case of numpy, I think cmake would be difficult to integrate into distutils > Yes, waf is pretty cool, even though the last time I looked at it, it > wasn't able to compile my project, which was larger than just couple > files. > I doubt this has anything to do with the project size. Waf seems more scalable than scons, and scons can handle big projects (almost all vmware projects are built using scons, for example), albeit slowly. > As to the other things, one nice thing about cmake is that it is > production ready right now, it is well tested (kde4) and it is in > distributions (e.g. Debian). For portability reasons, I think the build system should always distributed with the project you are building (like autoconf does, actually); this is another drawback of cmake. Waf's main author is against any waf packaging - waf can be distributed as one < 100 kb python file, which is very nice. I don't want to sound overly critical against cmake: obviously, since it is used by KDE4, it is definitely a good software. It supports some features which I wish scons had (rpath and install-relinking, distribution related features). David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 7:56 PM, David Cournapeau wrote: > Ondrej Certik wrote: >>> That's exactly what I don't like about cmake - it means you can't >>> produce accurate builds (you need to rerun cmake everytime you change >>> the configuration or dependencies, whereas this is automatic with >>> scons/waf). It also have (used to have) very poor documentation unless >>> you buy the book - but it looks like this is changing. >>> >> >> You can always rerun cmake if you want (and make it automatic). Imho >> that's not a problem. But maybe it is done in a better way with scons. >> > > I think it is a problem - it means you have to update it explicitly when > the configuration changes. In scons, the signature concept is quite > powerful: not only are file dependencies handled, but also command > lines, etc... For example, in numscons, if you change the blas/lapack > from atlas to say MKL, only linking and eventually configuration changes > are rebuilt. If you change the fortran compiler but not the C compiler, > only fortran code is rebuilt. All of this is 100 % automatic. I got an email from Alexander Neundorf from kde, and with his permission, I am forwarding it here, because he is not subscribed. As he writes below, cmake is in fact cabable of the above. -- Forwarded message -- From: Alexander Neundorf Date: Mon, Feb 9, 2009 at 1:28 PM Subject: cmake information To: Ondrej Certik Hi Ondrej, I hope you are the Ondrej from this thread: http://groups.google.com/group/Numpy-discussion/browse_thread/thread/c12f96b9ac367f57 If not, please let me know and I'll try to find the right email address. I saw this in the thread: --8<-8<8< Ondrej Certik wrote: >> That's exactly what I don't like about cmake - it means you can't >> produce accurate builds (you need to rerun cmake everytime you change >> the configuration or dependencies, whereas this is automatic with >> scons/waf). It also have (used to have) very poor documentation unless >> you buy the book - but it looks like this is changing. > You can always rerun cmake if you want (and make it automatic). Imho > that's not a problem. But maybe it is done in a better way with scons. I think it is a problem - it means you have to update it explicitly when the configuration changes. In scons, the signature concept is quite powerful: not only are file dependencies handled, but also command lines, etc... For example, in numscons, if you change the blas/lapack from atlas to say MKL, only linking and eventually configuration changes are rebuilt. If you change the fortran compiler but not the C compiler, only fortran code is rebuilt. All of this is 100 % automatic. --8<-8<8< CMake does handle this automatically. E.g. if include directories are changed (which you do by editing a CMakeLists.txt or the cmake cache), all files which are affected by the are rebuilt. If some library changes, everything linking to this library is linked again. If any of the files the build depends on (e.g. a CMakeLists.txt, or an included .cmake file, or the cmake cache) is changed, cmake is automatically rerun, and the makefiles/project files are regenerated. I don't know what happens if you are using both C and Fortran in one project and change one of them. I think this is actually not possible, you can (or should) not change the compiler of an existing build tree. Basically everything which uses this compiler is invalid then, the object files, the results of tests etc. I'm not sure handling this separately for different languages within one project is supported by cmake. Alex ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
> CMake does handle this automatically. > E.g. if include directories are changed (which you do by editing a > CMakeLists.txt or the cmake cache), all files which are affected by the are > rebuilt. If some library changes, everything linking to this library is > linked again. > If any of the files the build depends on (e.g. a CMakeLists.txt, or an > included .cmake file, or the cmake cache) is changed, cmake is automatically > rerun, and the makefiles/project files are regenerated. > > I don't know what happens if you are using both C and Fortran in one project > and change one of them. I think this is actually not possible, you can (or > should) not change the compiler of an existing build tree. Basically > everything which uses this compiler is invalid then, the object files, the > results of tests etc. I'm not sure handling this separately for different > languages within one project is supported by cmake. Is all of this handled just by calling make (after the initial cmake call), or do you have to first recall cmake and then make? Brian > Alex > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Mon, Feb 9, 2009 at 2:35 PM, Brian Granger wrote: >> CMake does handle this automatically. >> E.g. if include directories are changed (which you do by editing a >> CMakeLists.txt or the cmake cache), all files which are affected by the are >> rebuilt. If some library changes, everything linking to this library is >> linked again. >> If any of the files the build depends on (e.g. a CMakeLists.txt, or an >> included .cmake file, or the cmake cache) is changed, cmake is automatically >> rerun, and the makefiles/project files are regenerated. >> >> I don't know what happens if you are using both C and Fortran in one project >> and change one of them. I think this is actually not possible, you can (or >> should) not change the compiler of an existing build tree. Basically >> everything which uses this compiler is invalid then, the object files, the >> results of tests etc. I'm not sure handling this separately for different >> languages within one project is supported by cmake. > > Is all of this handled just by calling make (after the initial cmake > call), or do you have to first recall cmake and then make? I just tried that and it is handled automatically by calling "make". It's really cool! Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
