Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
Matthieu Brucher wrote: The 2.6 seems to use VC 2005 Express, I don't know about py3000(?), with associated upgrade issues. But what if the next MS compiler has again broken libc implementation ? (Incidently, VS2005 was not used for python2.5 for even more broken libc than in 2003): http://mail.python.org/pipermail/python-dev/2006-April/064126.html I don't what he meant by a broken libc, if it is the fact that there is a lot of deprecated standard functions, I don't call it broken (besides, this deprecation follows a technical paper that describe the new safe functions, although it does not deprecate these functions). If unilaterally deprecating standard functions which exist for years is not broken, I really wonder what is :) cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Hi,
Examples look good. It seems that you have lots of work ahead;) to add
numpy.distutils features that are required to build numpy/scipy.
Few comments:
1) Why SConstruct does not have extension? It looks like a python file
and .py extension could be used.
2) It seems that scons does not interfare with numpy.distutils much.
If this is true and numpy/scipy builds will not break when scons is
not installed then I think you could continue the scons support
development in trunk.
3) In future, if we are going to replace using distutils with scons
then all numpy/scipy need SConstruct scripts. I think one can implement
these scripts already now and use, say, setupscons.py, containing only
def configuration(parent_package='',top_path=None):
from numpy.distutils.misc_util import Configuration
config = Configuration('packagename',parent_package,top_path)
config.add_sconscript('SConstruct')
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(configuration=configuration)
to build the packages. Or can scons already be used with only
SConstruct script and setupscons.py are not needed? Implementing these
scripts now would give a good idea what features are required in using
scons to build numpy/scipy packages.
Also, it will prove that scons can replace numpy.distutils in future,
4) though, we cannot remove numpy.distutils for backward compatibility
with software using numpy.distutils. However, it should be possible
to enhance Configuration class to generate the corresponding SConstruct
scripts. It will save some work when converting configuration()
functions to SConstruct scripts.
Looking forward for not using distutils,
Pearu
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Pearu Peterson wrote:
Hi,
Examples look good. It seems that you have lots of work ahead;) to add
numpy.distutils features that are required to build numpy/scipy.
Hi Pearu,
Thanks for reviewing this, especially since you are arguably the
most knowledgeable about this part of numpy :)
Few comments:
1) Why SConstruct does not have extension? It looks like a python file
and .py extension could be used.
It looks like but isn't. This is actually the part of scons I like the
least: scons uses python, but sconscripts files are not proper python
modules. Sconscripts files are used declaratively:
http://www.scons.org/doc/HTML/scons-user/x348.html
I think this is one of the biggest design mistake of scons. Waf, which
was briefly mentioned by David M. Cooke, and which is based on scons,
does not do this, though (I did not use waf because it is new, is
maintained only by one developer, and is more geared toward unix, that
is MS tools are not supported).
2) It seems that scons does not interfare with numpy.distutils much.
If this is true and numpy/scipy builds will not break when scons is
not installed then I think you could continue the scons support
development in trunk.
It won't break if scons is not installed because scons sources are
copied into the branch. Scons developers explicitely support this:
http://www.scons.org/faq.php#SS_3_3
(AFAIK, it does not pose any problem license-wise, since scons is new
BSD license; it adds ~350 kb of compressed source code to numpy).
3) In future, if we are going to replace using distutils with scons
then all numpy/scipy need SConstruct scripts. I think one can implement
these scripts already now and use, say, setupscons.py, containing only
def configuration(parent_package='',top_path=None):
from numpy.distutils.misc_util import Configuration
config = Configuration('packagename',parent_package,top_path)
config.add_sconscript('SConstruct')
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(configuration=configuration)
to build the packages. Or can scons already be used with only
SConstruct script and setupscons.py are not needed? Implementing these
scripts now would give a good idea what features are required in using
scons to build numpy/scipy packages.
Also, it will prove that scons can replace numpy.distutils in future,
That's what I had in mind (setupscons.py). I don't see any big reasons
for not merging the current branch (since it is optional and should not
break anything).
Now, concerning migrating all compiled extensions to scons, I would
prefer avoiding doing it in the trunk; but I heard horror stories about
subversion and merging, so maybe staying outside the trunk is too risky
? Also, when I said that I don't see big problems to replace distutils
for compiled extensions, I lied by over-simplification. If scons is used
for compiled extension, with the current design, distutils will call
scons for each package. Calling scons is expensive (importing many
modules, etc...: this easily takes around one second for each non
trivial Sconscript), and also, because each of them is independent, we
may check several times for the same thing (fortran compiler, etc...),
which would really add up to the build time.
I see two approaches here:
- do not care about it because numpy is unlikely to become really bigger
- considering that numpy is really one big package (contrary to
scipy which, at least in my understanding, is gearing toward less
inter-package dependencies ?), we should only have one big sconscript
for configuration (checking blas/lapack, compilers, etc...), and use
scons recursively. In this case, it should not be much slower than the
current system.
4) though, we cannot remove numpy.distutils for backward compatibility
with software using numpy.distutils. However, it should be possible
to enhance Configuration class to generate the corresponding SConstruct
scripts. It will save some work when converting configuration()
functions to SConstruct scripts.
Could you elaborate on this point ? I am not sure what you mean by
generating Configuration class ?
cheers,
David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
David Cournapeau wrote: Pearu Peterson wrote: 2) It seems that scons does not interfare with numpy.distutils much. If this is true and numpy/scipy builds will not break when scons is not installed then I think you could continue the scons support development in trunk. It won't break if scons is not installed because scons sources are copied into the branch. Scons developers explicitely support this: http://www.scons.org/faq.php#SS_3_3 (AFAIK, it does not pose any problem license-wise, since scons is new BSD license; it adds ~350 kb of compressed source code to numpy). I think this is good. Does scons require python-dev? If not then this will solve one of the frequent issues that new users may experience: not installed distutils. Now, concerning migrating all compiled extensions to scons, I would prefer avoiding doing it in the trunk; but I heard horror stories about subversion and merging, so maybe staying outside the trunk is too risky ? I think numpy is quite stable now that it's safe to develop in a branch (if trunk is very actively developed then merging branches can be a nightmare). However, IMHO using a branch makes other developers to stay aside from branch development and in time it is more and more difficult to merge. Also, when I said that I don't see big problems to replace distutils for compiled extensions, I lied by over-simplification. If scons is used for compiled extension, with the current design, distutils will call scons for each package. Calling scons is expensive (importing many modules, etc...: this easily takes around one second for each non trivial Sconscript), and also, because each of them is independent, we may check several times for the same thing (fortran compiler, etc...), which would really add up to the build time. I see two approaches here: - do not care about it because numpy is unlikely to become really bigger - considering that numpy is really one big package (contrary to scipy which, at least in my understanding, is gearing toward less inter-package dependencies ?), we should only have one big sconscript for configuration (checking blas/lapack, compilers, etc...), and use scons recursively. In this case, it should not be much slower than the current system. Note that building a subpackage in subpackage directory must be supported. So a big sconscript may not be an option. The third approach would be to cache those checks that are called frequently to, say, $HOME/.numpy/scons. numpy.distutils setup.py script gathers the information from subpackage setup.py scripts recursively and then passes all the information to one setup function call. I think setupscons.py should do the same. If scons does not support recursive reading of scons scripts then the corresponding feature should be implemented, I guess it would not be difficult. 4) though, we cannot remove numpy.distutils for backward compatibility with software using numpy.distutils. However, it should be possible to enhance Configuration class to generate the corresponding SConstruct scripts. It will save some work when converting configuration() functions to SConstruct scripts. Could you elaborate on this point ? I am not sure what you mean by generating Configuration class ? I meant that Configuration class could have a method, say toscons(filename), that will generate SConstruct script from the information that Configuration instance holds. I thought that this would just ease creating SConstruct scripts from existing setup.py files. Pearu ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Pearu Peterson wrote: I think this is good. Does scons require python-dev? If not then this will solve one of the frequent issues that new users may experience: not installed distutils. Isn't distutils included in python library ? Anyway, scons does not require anything else than a python interpreter. Actually, an explicit requirement of scons is to support any python starting at version 1.5.2 (this is another important point which I consider important for a replacement of numpy.distutils). But please remember that this work is not intended at replacing distutils entirely, only the build_ext/build_lib commands. All the build process would still be driven by distutils (in particular, supporting distutils is a requirement to support setuptools and eggs). I think numpy is quite stable now that it's safe to develop in a branch (if trunk is very actively developed then merging branches can be a nightmare). However, IMHO using a branch makes other developers to stay aside from branch development and in time it is more and more difficult to merge. I don't have strong experience in subversion, so I was afraid of that. Do I understand correctly that you suggest opening a new dev branch, and then do all subsequent dev (including non distutils/scons related ones) there ? The third approach would be to cache those checks that are called frequently to, say, $HOME/.numpy/scons. numpy.distutils setup.py script gathers the information from subpackage setup.py scripts recursively and then passes all the information to one setup function call. I think setupscons.py should do the same. If scons does not support recursive reading of scons scripts then the corresponding feature should be implemented, I guess it would not be difficult. Scons supports recursive calls. To be more clear about the possibilities of scons wrt to this, let's take a simplified example: root/Sconstruct root/numpy/Sconscript root/numpy/core/Sconscript root/numpy/linalg/Sconscript If you are in root and call scons ( is a shell prompt): root scons Then it will call recursively all the sconscript (as long as you request it in the sconscript files). The other supported method is root/numpy/linalg scons -u this will look every parent directory to find the root Sconstruct. So as long as we configure everything (by everything, I mean checking for libraries and compilers) in the root Sconscript, this should work. To sum it up: the build process would be more like projects using autotools; a configure step, and a build step (this would be internal; there is no need to expose this mechanism to the user). I meant that Configuration class could have a method, say toscons(filename), that will generate SConstruct script from the information that Configuration instance holds. I thought that this would just ease creating SConstruct scripts from existing setup.py files. I don't think it would worth the effort for numpy (the main work really is to implement and test all the checkers: blas/lapack, fortran). Now, as a general migration tool, this may be useful. But since we would still use distutils, it would only be useful if it is easy to develop such as tool. cheers, David P.S: would it be easy for you to make a list of requirements for fortran ? By requirement, I mean things like name mangling and so on ? Something like the autoconf macros: http://www.gnu.org/software/autoconf/manual/autoconf-2.57/html_node/autoconf_66.html ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
I don't what he meant by a broken libc, if it is the fact that there is a lot of deprecated standard functions, I don't call it broken (besides, this deprecation follows a technical paper that describe the new safe functions, although it does not deprecate these functions). If unilaterally deprecating standard functions which exist for years is not broken, I really wonder what is :) They are deprecated (although a simple flag can get rid of those deprecation) not removed. Besides, the deprecated functions are in fact functions that can lead to security issues (for the first time Microsoft did something not completely stupid about this topic), so telling that the programmer should not use them but more secure one may be seen as a good idea (from a certain point of view). Matthieu ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
David Cournapeau wrote: Pearu Peterson wrote: I think this is good. Does scons require python-dev? If not then this will solve one of the frequent issues that new users may experience: not installed distutils. Isn't distutils included in python library ? Not always. For example, in debian one has to install python-dev separately from python. Anyway, scons does not require anything else than a python interpreter. Actually, an explicit requirement of scons is to support any python starting at version 1.5.2 (this is another important point which I consider important for a replacement of numpy.distutils). Though, it is irrelevant as numpy/scipy packages require python versions starting at 2.3. I think numpy is quite stable now that it's safe to develop in a branch (if trunk is very actively developed then merging branches can be a nightmare). However, IMHO using a branch makes other developers to stay aside from branch development and in time it is more and more difficult to merge. I don't have strong experience in subversion, so I was afraid of that. Do I understand correctly that you suggest opening a new dev branch, and then do all subsequent dev (including non distutils/scons related ones) there ? No, my original suggestion was that I don't mind if you would develop scons support in trunk as it does not affect the current state of numpy/scipy builds. Don't know if other developers would have objections in that. numpy.distutils setup.py script gathers the information from subpackage setup.py scripts recursively and then passes all the information to one setup function call. I think setupscons.py should do the same. If scons does not support recursive reading of scons scripts then the corresponding feature should be implemented, I guess it would not be difficult. Scons supports recursive calls. To be more clear about the possibilities of scons wrt to this, let's take a simplified example: root/Sconstruct root/numpy/Sconscript root/numpy/core/Sconscript root/numpy/linalg/Sconscript If you are in root and call scons ( is a shell prompt): root scons Then it will call recursively all the sconscript (as long as you request it in the sconscript files). The other supported method is root/numpy/linalg scons -u this will look every parent directory to find the root Sconstruct. My point was that root/numpy/linalg scons should work (without the -u option). A subpackage may not require all the stuff that other subpackages require and therefore scons should not configure everything - it's a waste of time and efforts - especially if something is broken in upper packages but not in the subpackage. I meant that Configuration class could have a method, say toscons(filename), that will generate SConstruct script from the information that Configuration instance holds. I thought that this would just ease creating SConstruct scripts from existing setup.py files. I don't think it would worth the effort for numpy (the main work really is to implement and test all the checkers: blas/lapack, fortran). Now, as a general migration tool, this may be useful. But since we would still use distutils, it would only be useful if it is easy to develop such as tool. Yes, that's a low priority feature. P.S: would it be easy for you to make a list of requirements for fortran ? By requirement, I mean things like name mangling and so on ? Something like the autoconf macros: http://www.gnu.org/software/autoconf/manual/autoconf-2.57/html_node/autoconf_66.html To use f2py succesfully, a fortran compiler must support flags that make fortran symbol names lowercase and with exactly one underscore at the end of a name. This is required when using numpy.distutils. f2py generated modules make use of the following CPP macros: -DPREPEND_FORTRAN -DNO_APPEND_FORTRAN -DUPPERCASE_FORTRAN -DUNDERSCORE_G77 and therefore the above requirement would not be needed if scons could figure out how some particular compiler mangles the names of fortran symbols. This would be especially useful since some fortran compiler vendors change the compiler flags between compiler versions and one has to update numpy.distutils files accordingly. Note that using hardcoded name mangeling flags may be still required for certian Fortran 90 compilers (which ones exactly, I don't member now) that by default produce symbol names with special characters like $ or . for F90 modules and making these names unaccessible to C programs. Pearu ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] appending extra items to arrays
On 10/10/07, Anne Archibald [EMAIL PROTECTED] wrote:
On 11/10/2007, Robert Kern [EMAIL PROTECTED] wrote:
Appending to a list then converting the list to an array is the most
straightforward way to do it. If the performance of this isn't a
problem, I
recommend leaving it alone.
Just a speculation:
Python strings have a similar problem - they're immutable, and so are
even more resistant to growth than numpy arrays. For those situations
where you really really want to grow a srting, python provides
StringIO, where you keep efficiently adding to the string, then
finalize it to get the real string out. Would something analogous be
interesting for arrays?
Have you looked at array.array?
import array
a = array.array('i')
a.append(2)
a.append(7)
a
array('i', [2, 7])
--
. __
. |-\
.
. [EMAIL PROTECTED]
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] appending extra items to arrays
On 11/10/2007, Robert Kern [EMAIL PROTECTED] wrote: Appending to a list then converting the list to an array is the most straightforward way to do it. If the performance of this isn't a problem, I recommend leaving it alone. Thanks, I'll leave it as is - I was just wondering if there was a better way to do it. Cheers Adam ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] diff of find of diff trick
Basically, what you want to do is a histogram. Numpy has that
functionality built in. However: the version built in to numpy is about
as suboptimal as yours. The problem is the unnecessary sorting of the data.
In principle, a histogram does not need any sorting, so it could be done
in strictly O(N). I don't see any simply way of doing so efficiently in
numpy without resorting to a costly loop. We had a discussion about this
a while ago but I cannot remember any resolution. The proper thing to do
would probably be a C routine.
If performance is crucial to you, I would suggest using weave. A minimal
example as starting point is given below.
Greetings,
Norbert
---
#!/usr/bin/env python
import numpy
import scipy.weave
import scipy.weave.converters
def histogram(data, nR):
res = numpy.zeros(nR,int)
dataflat = data.flatten()
Ndata = len(dataflat)
scipy.weave.inline(
for(int i=0;iNdata;i++) {
int d = dataflat(i);
if(d nR d = 0) {
res(d) += 1;
}
}
,
arg_names = [dataflat,res,Ndata,nR],
type_converters = scipy.weave.converters.blitz,
)
return res
a = numpy.array([0,3,1,2,3,5,2,3,2,4,5,1,3,4,2])
print a
print histogram(a,4)
---
Robin wrote:
Hello,
As a converting MATLAB user I am bringing over some of my code. A core
function I need is to sample the probability distribution of a given
set of data. Sometimes these sets are large so I would like to make it
as efficient as possible. (the data are integers representing members
of a discrete space)
In MATLAB the best way I found was the diff of find of diff trick
which resulted in the completely vectorised solution (below). Does it
make sense to translate this into numpy? I don't have a feel yet for
what is fast/slow - are the functions below built in and so quick (I
thought diff might not be).
Is there a better/more pythonic way to do it?
function Pr=prob(data, nR)
Pr= zeros(nR,1);
% diff of find of diff trick for counting number of elements
temp = sort(data(data0));% want vector excluding P(0)
dtemp= diff([temp;max(temp)+1]);
count = diff(find([1;dtemp]));
indx = temp(dtemp0);
Pr(indx)= count ./ numel(data);% probability
Thanks
Robin
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] appending extra items to arrays
If you do not know the size of your array before you finalize it, then
you should use lists whenever you can. I just cooked up a short
example:
##
import timeit
import numpy as N
values = range(1)
def appendArray(values):
result = N.array([], dtype=int)
for value in values:
result = N.append(result, value)
return result
def appendList(values):
result = []
for value in values:
result.append(value)
return N.array(result)
test = timeit.Timer('appendArray(values)',
'from __main__ import appendArray, values')
t1 = test.timeit(number=10)
test2 = timeit.Timer('appendList(values)',
'from __main__ import appendList, values')
t2 = test2.timeit(number=10)
print Total Time with array: + str(t1)
print Total Time with list: + str(t2)
# Result #
Total Time with array: 2.12951189331
Total Time with list: 0.0469707035741
Hope this helps,
MJ
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Adam Mercer
Sent: Thursday, October 11, 2007 7:42 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] appending extra items to arrays
On 11/10/2007, Robert Kern [EMAIL PROTECTED] wrote:
Appending to a list then converting the list to an array is the most
straightforward way to do it. If the performance of this isn't a
problem, I
recommend leaving it alone.
Thanks, I'll leave it as is - I was just wondering if there was a
better way to do it.
Cheers
Adam
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Pearu Peterson wrote: David Cournapeau wrote: Pearu Peterson wrote: I think this is good. Does scons require python-dev? If not then this will solve one of the frequent issues that new users may experience: not installed distutils. Isn't distutils included in python library ? Not always. For example, in debian one has to install python-dev separately from python. You'd still need python-dev(el) for the headers if not distutils. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] appending extra items to arrays
On 11/10/2007, Mark Janikas [EMAIL PROTECTED] wrote: If you do not know the size of your array before you finalize it, then you should use lists whenever you can. I just cooked up a short example: snip # Result # Total Time with array: 2.12951189331 Total Time with list: 0.0469707035741 Hope this helps, That is helpful, I thought that using arrays would be much faster but its clearly not in this case. Thanks Adam ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.distutils.cpuinfo bugs?
On Tue, 2 Oct 2007, David M. Cooke apparently wrote: Should be fixed now. The update seems to work for all my students who were having problems. Thank you, Alan Isaac ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [OT]numpy/Windows shared arrays between processes?
Matthieu Brucher wrote: I don't what he meant by a broken libc, if it is the fact that there is a lot of deprecated standard functions, I don't call it broken (besides, this deprecation follows a technical paper that describe the new safe functions, although it does not deprecate these functions). If unilaterally deprecating standard functions which exist for years is not broken, I really wonder what is :) They are deprecated (although a simple flag can get rid of those deprecation) not removed. deprecated means that they will disappear soon, and (worse), people will chose to use the non deprecated, hence programming non portable code (those functions are neither portable or standard; different OS have different implementation). Besides, the deprecated functions are in fact functions that can lead to security issues This is far from obvious. Actually, numerous people (who I trust more than MS) have raised their concern on those functions: they add complexity for no obvious advantage. http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1106.txt cheers, Ddavid ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] For review: first milestone of scons support in numpy
Pearu Peterson wrote: Anyway, scons does not require anything else than a python interpreter. Actually, an explicit requirement of scons is to support any python starting at version 1.5.2 (this is another important point which I consider important for a replacement of numpy.distutils). Though, it is irrelevant as numpy/scipy packages require python versions starting at 2.3. Sure, I just meant that we don't have to worry that scons may depend on 2.4 or 2.5 features (I am not sure waf have such a requirement, for example). No, my original suggestion was that I don't mind if you would develop scons support in trunk as it does not affect the current state of numpy/scipy builds. Don't know if other developers would have objections in that. Ah, ok. I am not sure I can guarantee it will never affect the build process. As this is a pretty fundamental change, I though about doing that in a 1.1 branch of numpy. But I do not have strong opinion on that, nor much experience in that area. My point was that root/numpy/linalg scons should work (without the -u option). Since scons is called by distutils, and never by the user, I don't see how this can be a problem ? distutils would have to figure out when to use -u, not the user. A subpackage may not require all the stuff that other subpackages require and therefore scons should not configure everything - it's a waste of time and efforts - especially if something is broken in upper packages but not in the subpackage. Note that scons caches the configuration, so I don't think speed will be an issue here (except maybe if you keep changing the configuration before building). What I could do is for each subpackage, to declare the tests to use in each subpackage, but then, what to do if two packages have some common tests ? I cannot just remove double tests, because the order is significant (because of link options, for example). The way I see it, either we keep the current behaviour (each package is totally independant, scons is called for each subpackage, and scons has no way to know about other subpackages; this has the disadvantage of being slower: this will be significant of many subpackages use costly checks like fortran mangling and so on), or we have a main sconscript with the configuration, which does not prevent building subpackages, but which requires a global configuration. If none of those approach seems right to you, I will see if I can come up with something better, but this will certainly add some complexity (or I am just stupid to see an obvious solution :) ). To use f2py succesfully, a fortran compiler must support flags that make fortran symbol names lowercase and with exactly one underscore at the end of a name. This is required when using numpy.distutils. f2py generated modules make use of the following CPP macros: -DPREPEND_FORTRAN -DNO_APPEND_FORTRAN -DUPPERCASE_FORTRAN -DUNDERSCORE_G77 and therefore the above requirement would not be needed if scons could figure out how some particular compiler mangles the names of fortran symbols. This would be especially useful since some fortran compiler vendors change the compiler flags between compiler versions and one has to update numpy.distutils files accordingly. Thank you for those information. Do I understand correctly (sorry for being slow, but I don't know anything about fortran) that what you need is macro like: - AC_F77_WRAPPERS (which defines C macros for converting C functions to fortran compiler mangling) - AC_F77_FUNC (which retrieves the name mangled by fortran linked from the 'canonical' name: this is already implemented, as it is necessary for checking blas/lapack) And that's it ? cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
