Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Pearu Peterson wrote:
Yes. I was considering the situations where distutils in not used at
all. May be it will be relevant in future..
If distutils is not used at all, there will be more difficult problems
to solve than this one, then. I think it is safe to assume that
distutils will stay there for quite some time, no ?
Note that I try hard in numpy/scons to separate the scons specific from
the parts which are not specific to scons. If in the future, we decide
to go on the waf route, or whatever, I hope that most of the logic would
be easily reused: for example, parsing the fortran link output to guess
the link options for C/Fortran is totally independant from scons (this
has the nice advantage of making regression tests possible on all
platforms).
I would prefer the former solution where each package is totally
independent. Let's first see what is the acctual performance and then
maybe consider more complex solutions if the build is really slow.
I measured the initialization time of scons if called for every
subpackage in numpy/: this adds less than two percents of total build
time, so this is not as bad as I feared (specially since only a few
package would need scons: pure python ones do not need them). And scons
can also give some speed advantage, because it explicitely support in a
totally safe way parallel builds.
So basically, I went to the same conclusion than you. I have started
porting numpy.core to scons (by far the most difficult subpackage, at
least a priori) following this route.
These macros are fine. May be we need more such macros for F90 compilers
but we can discuss about these when the corresponding issues will emerge.
Oh, I forgot to say it, but the macro would be implemented for both F77
and F90 (and F2003 or whatever should we need it). As most of the logic
is independent of the version used, this is not a problem.
However, if you could also provide the macros
that f2py uses then everything would work with no change in f2py part.
f2py uses the following defintion of F_FUNC, for example:
I can of course change the name of the macro (this could be an option,
even). I guess it is better to follow f2py convention than autotools
here, in our case.
system_info has a feature of providing architecture specific
and optimization flags for fortran compilers. Could scons
support such a feature?
scons uses the concept of Environment objects: they have sensible
defaults (compiler/platform combination), but you can override anything
you want, and they are used for compilation.
env = Environment()
env.AppendUnique(F77FLAGS = -O3)
env.Program('hello', source = ['hello.f'])
Of course, in our case, we should provide some support such as instead
of adding -O3, we have something like a FULL_OPTIMIZED variable which
contains the optimization flags tweaked for a particular
compiler/platform. Better, we can check that a given flag is actually
supported. This is something I have not done yet, but that's the kind of
things scons does really well, so I don't see any problems here.
The main problem right now is to convert all the checks in
numpy/core/setup.py, which is the only non trivial setup.py file as far
as I can tell in numpy. This will take a few hours to get it right on
major platforms :)
cheers,
David
P.S If you have access to some fortran compiler you would like to check,
you can check my bzr branch on launchpad (this is done outside numpy
because some things may be merged to scons sources at some point):
https://code.launchpad.net/~david-ar/numpy.scons.support/dev
And test it by: scons CC=ccompiler F77=f77compiler F90=f90compiler. For
example: scons CC=icc F77=g77 F90=ifort.
(If you change compiler, you may need to add the option --config=force).
This is a fake sconscript which tests the following:
- does the fortran compiler works (e.g. can compile a dummy program)
- find the dummy main
- find the verbose flag (useful to find informations at runtime)
- find the name mangling
- find the necessary options for C/Fortran mix (only implemented for
F77 for now).
There is a pretty good chance that you will be able to break it (in
particular finding the options for C/Fortran: I have not yet implemented
all the necessary parsing), but in this case, please provides me with
the config.log (which logs all the configuration dones by scons).
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Hi,
Examples look good. It seems that you have lots of work ahead;) to add
numpy.distutils features that are required to build numpy/scipy.
Few comments:
1) Why SConstruct does not have extension? It looks like a python file
and .py extension could be used.
2) It seems that scons does not interfare with numpy.distutils much.
If this is true and numpy/scipy builds will not break when scons is
not installed then I think you could continue the scons support
development in trunk.
3) In future, if we are going to replace using distutils with scons
then all numpy/scipy need SConstruct scripts. I think one can implement
these scripts already now and use, say, setupscons.py, containing only
def configuration(parent_package='',top_path=None):
from numpy.distutils.misc_util import Configuration
config = Configuration('packagename',parent_package,top_path)
config.add_sconscript('SConstruct')
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(configuration=configuration)
to build the packages. Or can scons already be used with only
SConstruct script and setupscons.py are not needed? Implementing these
scripts now would give a good idea what features are required in using
scons to build numpy/scipy packages.
Also, it will prove that scons can replace numpy.distutils in future,
4) though, we cannot remove numpy.distutils for backward compatibility
with software using numpy.distutils. However, it should be possible
to enhance Configuration class to generate the corresponding SConstruct
scripts. It will save some work when converting configuration()
functions to SConstruct scripts.
Looking forward for not using distutils,
Pearu
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Pearu Peterson wrote:
Hi,
Examples look good. It seems that you have lots of work ahead;) to add
numpy.distutils features that are required to build numpy/scipy.
Hi Pearu,
Thanks for reviewing this, especially since you are arguably the
most knowledgeable about this part of numpy :)
Few comments:
1) Why SConstruct does not have extension? It looks like a python file
and .py extension could be used.
It looks like but isn't. This is actually the part of scons I like the
least: scons uses python, but sconscripts files are not proper python
modules. Sconscripts files are used declaratively:
http://www.scons.org/doc/HTML/scons-user/x348.html
I think this is one of the biggest design mistake of scons. Waf, which
was briefly mentioned by David M. Cooke, and which is based on scons,
does not do this, though (I did not use waf because it is new, is
maintained only by one developer, and is more geared toward unix, that
is MS tools are not supported).
2) It seems that scons does not interfare with numpy.distutils much.
If this is true and numpy/scipy builds will not break when scons is
not installed then I think you could continue the scons support
development in trunk.
It won't break if scons is not installed because scons sources are
copied into the branch. Scons developers explicitely support this:
http://www.scons.org/faq.php#SS_3_3
(AFAIK, it does not pose any problem license-wise, since scons is new
BSD license; it adds ~350 kb of compressed source code to numpy).
3) In future, if we are going to replace using distutils with scons
then all numpy/scipy need SConstruct scripts. I think one can implement
these scripts already now and use, say, setupscons.py, containing only
def configuration(parent_package='',top_path=None):
from numpy.distutils.misc_util import Configuration
config = Configuration('packagename',parent_package,top_path)
config.add_sconscript('SConstruct')
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(configuration=configuration)
to build the packages. Or can scons already be used with only
SConstruct script and setupscons.py are not needed? Implementing these
scripts now would give a good idea what features are required in using
scons to build numpy/scipy packages.
Also, it will prove that scons can replace numpy.distutils in future,
That's what I had in mind (setupscons.py). I don't see any big reasons
for not merging the current branch (since it is optional and should not
break anything).
Now, concerning migrating all compiled extensions to scons, I would
prefer avoiding doing it in the trunk; but I heard horror stories about
subversion and merging, so maybe staying outside the trunk is too risky
? Also, when I said that I don't see big problems to replace distutils
for compiled extensions, I lied by over-simplification. If scons is used
for compiled extension, with the current design, distutils will call
scons for each package. Calling scons is expensive (importing many
modules, etc...: this easily takes around one second for each non
trivial Sconscript), and also, because each of them is independent, we
may check several times for the same thing (fortran compiler, etc...),
which would really add up to the build time.
I see two approaches here:
- do not care about it because numpy is unlikely to become really bigger
- considering that numpy is really one big package (contrary to
scipy which, at least in my understanding, is gearing toward less
inter-package dependencies ?), we should only have one big sconscript
for configuration (checking blas/lapack, compilers, etc...), and use
scons recursively. In this case, it should not be much slower than the
current system.
4) though, we cannot remove numpy.distutils for backward compatibility
with software using numpy.distutils. However, it should be possible
to enhance Configuration class to generate the corresponding SConstruct
scripts. It will save some work when converting configuration()
functions to SConstruct scripts.
Could you elaborate on this point ? I am not sure what you mean by
generating Configuration class ?
cheers,
David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
David Cournapeau wrote: Pearu Peterson wrote: 2) It seems that scons does not interfare with numpy.distutils much. If this is true and numpy/scipy builds will not break when scons is not installed then I think you could continue the scons support development in trunk. It won't break if scons is not installed because scons sources are copied into the branch. Scons developers explicitely support this: http://www.scons.org/faq.php#SS_3_3 (AFAIK, it does not pose any problem license-wise, since scons is new BSD license; it adds ~350 kb of compressed source code to numpy). I think this is good. Does scons require python-dev? If not then this will solve one of the frequent issues that new users may experience: not installed distutils. Now, concerning migrating all compiled extensions to scons, I would prefer avoiding doing it in the trunk; but I heard horror stories about subversion and merging, so maybe staying outside the trunk is too risky ? I think numpy is quite stable now that it's safe to develop in a branch (if trunk is very actively developed then merging branches can be a nightmare). However, IMHO using a branch makes other developers to stay aside from branch development and in time it is more and more difficult to merge. Also, when I said that I don't see big problems to replace distutils for compiled extensions, I lied by over-simplification. If scons is used for compiled extension, with the current design, distutils will call scons for each package. Calling scons is expensive (importing many modules, etc...: this easily takes around one second for each non trivial Sconscript), and also, because each of them is independent, we may check several times for the same thing (fortran compiler, etc...), which would really add up to the build time. I see two approaches here: - do not care about it because numpy is unlikely to become really bigger - considering that numpy is really one big package (contrary to scipy which, at least in my understanding, is gearing toward less inter-package dependencies ?), we should only have one big sconscript for configuration (checking blas/lapack, compilers, etc...), and use scons recursively. In this case, it should not be much slower than the current system. Note that building a subpackage in subpackage directory must be supported. So a big sconscript may not be an option. The third approach would be to cache those checks that are called frequently to, say, $HOME/.numpy/scons. numpy.distutils setup.py script gathers the information from subpackage setup.py scripts recursively and then passes all the information to one setup function call. I think setupscons.py should do the same. If scons does not support recursive reading of scons scripts then the corresponding feature should be implemented, I guess it would not be difficult. 4) though, we cannot remove numpy.distutils for backward compatibility with software using numpy.distutils. However, it should be possible to enhance Configuration class to generate the corresponding SConstruct scripts. It will save some work when converting configuration() functions to SConstruct scripts. Could you elaborate on this point ? I am not sure what you mean by generating Configuration class ? I meant that Configuration class could have a method, say toscons(filename), that will generate SConstruct script from the information that Configuration instance holds. I thought that this would just ease creating SConstruct scripts from existing setup.py files. Pearu ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Pearu Peterson wrote: I think this is good. Does scons require python-dev? If not then this will solve one of the frequent issues that new users may experience: not installed distutils. Isn't distutils included in python library ? Anyway, scons does not require anything else than a python interpreter. Actually, an explicit requirement of scons is to support any python starting at version 1.5.2 (this is another important point which I consider important for a replacement of numpy.distutils). But please remember that this work is not intended at replacing distutils entirely, only the build_ext/build_lib commands. All the build process would still be driven by distutils (in particular, supporting distutils is a requirement to support setuptools and eggs). I think numpy is quite stable now that it's safe to develop in a branch (if trunk is very actively developed then merging branches can be a nightmare). However, IMHO using a branch makes other developers to stay aside from branch development and in time it is more and more difficult to merge. I don't have strong experience in subversion, so I was afraid of that. Do I understand correctly that you suggest opening a new dev branch, and then do all subsequent dev (including non distutils/scons related ones) there ? The third approach would be to cache those checks that are called frequently to, say, $HOME/.numpy/scons. numpy.distutils setup.py script gathers the information from subpackage setup.py scripts recursively and then passes all the information to one setup function call. I think setupscons.py should do the same. If scons does not support recursive reading of scons scripts then the corresponding feature should be implemented, I guess it would not be difficult. Scons supports recursive calls. To be more clear about the possibilities of scons wrt to this, let's take a simplified example: root/Sconstruct root/numpy/Sconscript root/numpy/core/Sconscript root/numpy/linalg/Sconscript If you are in root and call scons ( is a shell prompt): root scons Then it will call recursively all the sconscript (as long as you request it in the sconscript files). The other supported method is root/numpy/linalg scons -u this will look every parent directory to find the root Sconstruct. So as long as we configure everything (by everything, I mean checking for libraries and compilers) in the root Sconscript, this should work. To sum it up: the build process would be more like projects using autotools; a configure step, and a build step (this would be internal; there is no need to expose this mechanism to the user). I meant that Configuration class could have a method, say toscons(filename), that will generate SConstruct script from the information that Configuration instance holds. I thought that this would just ease creating SConstruct scripts from existing setup.py files. I don't think it would worth the effort for numpy (the main work really is to implement and test all the checkers: blas/lapack, fortran). Now, as a general migration tool, this may be useful. But since we would still use distutils, it would only be useful if it is easy to develop such as tool. cheers, David P.S: would it be easy for you to make a list of requirements for fortran ? By requirement, I mean things like name mangling and so on ? Something like the autoconf macros: http://www.gnu.org/software/autoconf/manual/autoconf-2.57/html_node/autoconf_66.html ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
David Cournapeau wrote: Pearu Peterson wrote: I think this is good. Does scons require python-dev? If not then this will solve one of the frequent issues that new users may experience: not installed distutils. Isn't distutils included in python library ? Not always. For example, in debian one has to install python-dev separately from python. Anyway, scons does not require anything else than a python interpreter. Actually, an explicit requirement of scons is to support any python starting at version 1.5.2 (this is another important point which I consider important for a replacement of numpy.distutils). Though, it is irrelevant as numpy/scipy packages require python versions starting at 2.3. I think numpy is quite stable now that it's safe to develop in a branch (if trunk is very actively developed then merging branches can be a nightmare). However, IMHO using a branch makes other developers to stay aside from branch development and in time it is more and more difficult to merge. I don't have strong experience in subversion, so I was afraid of that. Do I understand correctly that you suggest opening a new dev branch, and then do all subsequent dev (including non distutils/scons related ones) there ? No, my original suggestion was that I don't mind if you would develop scons support in trunk as it does not affect the current state of numpy/scipy builds. Don't know if other developers would have objections in that. numpy.distutils setup.py script gathers the information from subpackage setup.py scripts recursively and then passes all the information to one setup function call. I think setupscons.py should do the same. If scons does not support recursive reading of scons scripts then the corresponding feature should be implemented, I guess it would not be difficult. Scons supports recursive calls. To be more clear about the possibilities of scons wrt to this, let's take a simplified example: root/Sconstruct root/numpy/Sconscript root/numpy/core/Sconscript root/numpy/linalg/Sconscript If you are in root and call scons ( is a shell prompt): root scons Then it will call recursively all the sconscript (as long as you request it in the sconscript files). The other supported method is root/numpy/linalg scons -u this will look every parent directory to find the root Sconstruct. My point was that root/numpy/linalg scons should work (without the -u option). A subpackage may not require all the stuff that other subpackages require and therefore scons should not configure everything - it's a waste of time and efforts - especially if something is broken in upper packages but not in the subpackage. I meant that Configuration class could have a method, say toscons(filename), that will generate SConstruct script from the information that Configuration instance holds. I thought that this would just ease creating SConstruct scripts from existing setup.py files. I don't think it would worth the effort for numpy (the main work really is to implement and test all the checkers: blas/lapack, fortran). Now, as a general migration tool, this may be useful. But since we would still use distutils, it would only be useful if it is easy to develop such as tool. Yes, that's a low priority feature. P.S: would it be easy for you to make a list of requirements for fortran ? By requirement, I mean things like name mangling and so on ? Something like the autoconf macros: http://www.gnu.org/software/autoconf/manual/autoconf-2.57/html_node/autoconf_66.html To use f2py succesfully, a fortran compiler must support flags that make fortran symbol names lowercase and with exactly one underscore at the end of a name. This is required when using numpy.distutils. f2py generated modules make use of the following CPP macros: -DPREPEND_FORTRAN -DNO_APPEND_FORTRAN -DUPPERCASE_FORTRAN -DUNDERSCORE_G77 and therefore the above requirement would not be needed if scons could figure out how some particular compiler mangles the names of fortran symbols. This would be especially useful since some fortran compiler vendors change the compiler flags between compiler versions and one has to update numpy.distutils files accordingly. Note that using hardcoded name mangeling flags may be still required for certian Fortran 90 compilers (which ones exactly, I don't member now) that by default produce symbol names with special characters like $ or . for F90 modules and making these names unaccessible to C programs. Pearu ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Pearu Peterson wrote: David Cournapeau wrote: Pearu Peterson wrote: I think this is good. Does scons require python-dev? If not then this will solve one of the frequent issues that new users may experience: not installed distutils. Isn't distutils included in python library ? Not always. For example, in debian one has to install python-dev separately from python. You'd still need python-dev(el) for the headers if not distutils. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Pearu Peterson wrote: Anyway, scons does not require anything else than a python interpreter. Actually, an explicit requirement of scons is to support any python starting at version 1.5.2 (this is another important point which I consider important for a replacement of numpy.distutils). Though, it is irrelevant as numpy/scipy packages require python versions starting at 2.3. Sure, I just meant that we don't have to worry that scons may depend on 2.4 or 2.5 features (I am not sure waf have such a requirement, for example). No, my original suggestion was that I don't mind if you would develop scons support in trunk as it does not affect the current state of numpy/scipy builds. Don't know if other developers would have objections in that. Ah, ok. I am not sure I can guarantee it will never affect the build process. As this is a pretty fundamental change, I though about doing that in a 1.1 branch of numpy. But I do not have strong opinion on that, nor much experience in that area. My point was that root/numpy/linalg scons should work (without the -u option). Since scons is called by distutils, and never by the user, I don't see how this can be a problem ? distutils would have to figure out when to use -u, not the user. A subpackage may not require all the stuff that other subpackages require and therefore scons should not configure everything - it's a waste of time and efforts - especially if something is broken in upper packages but not in the subpackage. Note that scons caches the configuration, so I don't think speed will be an issue here (except maybe if you keep changing the configuration before building). What I could do is for each subpackage, to declare the tests to use in each subpackage, but then, what to do if two packages have some common tests ? I cannot just remove double tests, because the order is significant (because of link options, for example). The way I see it, either we keep the current behaviour (each package is totally independant, scons is called for each subpackage, and scons has no way to know about other subpackages; this has the disadvantage of being slower: this will be significant of many subpackages use costly checks like fortran mangling and so on), or we have a main sconscript with the configuration, which does not prevent building subpackages, but which requires a global configuration. If none of those approach seems right to you, I will see if I can come up with something better, but this will certainly add some complexity (or I am just stupid to see an obvious solution :) ). To use f2py succesfully, a fortran compiler must support flags that make fortran symbol names lowercase and with exactly one underscore at the end of a name. This is required when using numpy.distutils. f2py generated modules make use of the following CPP macros: -DPREPEND_FORTRAN -DNO_APPEND_FORTRAN -DUPPERCASE_FORTRAN -DUNDERSCORE_G77 and therefore the above requirement would not be needed if scons could figure out how some particular compiler mangles the names of fortran symbols. This would be especially useful since some fortran compiler vendors change the compiler flags between compiler versions and one has to update numpy.distutils files accordingly. Thank you for those information. Do I understand correctly (sorry for being slow, but I don't know anything about fortran) that what you need is macro like: - AC_F77_WRAPPERS (which defines C macros for converting C functions to fortran compiler mangling) - AC_F77_FUNC (which retrieves the name mangled by fortran linked from the 'canonical' name: this is already implemented, as it is necessary for checking blas/lapack) And that's it ? cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] For review: first milestone of scons support in numpy
Hi, (if you are not interested in numpy developement, you can stop now :) ). Following the discussion a few days ago on using scons to build extensions in numpy, I have reached a somewhat usable milestone, in the numpy.scons branch of numpy, and would like to hear some comments, remarks, critics, etc...: Where to get/see: - svn repository : http://svn.scipy.org/svn/numpy/branches/numpy.scons looking at the code: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons Examples: - To see how it feels from the package developer point of view, I have put three really simple examples: - Building a python extension: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/scons_fake/pyext - Building a ctypes-based package: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/scons_fake/ctypesext - An example on how to check for libraries and symbols in them: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/scons_fake/pyext For the numpy user, this should be totally transparent (no difference when building/installing). What: - This first milestone implements the following: - adding a scons command to numpy.distutils - adding an add_sconscript function to numpy.distutils setup, for packages willing to use scons - two builders: one for ctypes extension, and one for python extension - a basic implementation to check for libraries (the paths can be overwritten exactly like with distutils, using site.cfg; I have not yet implemented overwriting with environment variables). I have been testing this on the following platforms: - linux with gcc - linux with icc - linux with suncc - windows with MS toolikit 2003 - solaris studio express with suncc - mac os X (tiger, x86) And now ? - As discussed previously, I think numpy would benefit from exclusively using scons to build compiled extensions. I have started working on fortran support for scons (separate project, since this may be useful to all scons users, not just numpy): https://launchpad.net/numpy.scons.support and I can already do some non trivial things, not possible with numpy.distutils (automatically figuring out fortran mangling, flags for linking with C, blas/lapack flags). As expected, this is much more robust than distutils approach of hardcoding everything: although I used g77 for development, it worked without any change with ifort, gfortran and sun fortran compiler (on linux). There are still some issues for sure, but I don't see big problems. I don't want to do the work for nothing, though, so I would like to know the feeling of numpy developers first on this direction, in particular which platforms should work before merging consideration, etc... cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
