Re: [Numpy-discussion] NumPy Governance
On Mon, Dec 5, 2011 at 4:22 AM, Perry Greenfield pe...@stsci.edu wrote: I'm not sure I'm crazy about leaving final decision making for a board. A board may be a good way of carefully considering the issues, and it could make it's own recommendation (with a sufficient majority). But in the end I think one person needs to decide (and that decision may go against the board consensus, presumably only rarely). Why shouldn't that person be you? I haven't contributed to NumPy directly. But I can offer my experience with SymPy. I agree with Perry. Having one person being in charge as the last call (project leader) works excellent in my experience. For SymPy, that person has been me, up until a year ago (when I realized that I am too busy to do a good job as a project leader), when I passed it to Aaron Meurer. We always try to reach consensus, and the project leader's main job is to encourage such discussion. When consensus cannot be reached, he needs to make the decision (that happened maybe once or twice in the last 5 years and it is very rare). There seems to be quite strong community ownership in SymPy (that was Stefan's objection). I think the reason being that in fact we probably have something like a board of members, except that it is informal and it simply consists of people whose opinions the project leader highly values. And I think that it is very easy for anybody who gets involved with SymPy development to become trusted and thus his or her opinion will count. As such, for NumPy I think by default the project leader is Travis, who created it. He became busy in the last few years and so he could appoint a person, who will be the project leader. The list of possible people seems quite simple, I would choose somebody who is involved a lot with NumPy in the last 1 year (let's say): $ git shortlog -ns --since=1 year ago | head 651 Mark Wiebe 137 Charles Harris 72 David Cournapeau 61 Ralf Gommers 52 rgommers 29 Pearu Peterson 17 Pauli Virtanen 11 Chris Jordan-Squire 11 Matthew Brett 10 Christopher L. Farrow So anybody from the top 5 or 10 people seems ok. This has to be a personal decision, and I don't know what the actual contribution and involvement (and personal ability to be a project leader) is of the above people, so that's why it should be done by Travis (possibly consulting with somebody who he trusts and who is involved). For SymPy, here is the list from the 1 year ago when I passed the project leadership: $ git shortlog -ns --since=January 2010 --until January 2011 | head 317 Øyvind Jensen 150 Mateusz Paprocki 93 Aaron Meurer 81 Addison Cugini 79 Brian E. Granger 64 Ronan Lamy 61 Matt Curry 58 Ondřej Čertík 36 Chris Smith 34 Christian Muise It's not exactly accurate, as some of the branches from 2010 were merged in 2011, but it gives you a picture. The above list doesn't tell you who the best person should be. I knew that Aaron would be the best choice, and I consulted it with several core developers to see what the community thinks, and everybody told me, that if I need to pass it on, Aaron would be the choice. Since this was the first time for me doing this, I simply stated, that Aaron is the project leader from now on. And in couple months we clarified it a little bit, that I am the owner, in a sense that I own the domain and some servers and other things and I am ultimately responsible for the project (and I still have a say in non-coding related issues, like Google Summer of Code and such). For anything code related, Aaron has the last word, and I will not override it. The precise email is here: https://groups.google.com/d/topic/sympy/i-XD15syvqs/discussion You can compare it to today's list: $ git shortlog -ns --since=1 year ago | head 805Chris Smith 583 Mateusz Paprocki 508 Aaron Meurer 183 Ronan Lamy 150 Saptarshi Mandal 112 Tom Bachmann 101 Vladimir Perić 93 Gilbert Gede 91 Ondřej Čertík 89 Brian E. Granger So the activity has gone up after I stopped being the bottleneck, and after there was again a clear person, who is in charge and has time for it. Anyway, I just wanted to offer some experience that I gained with SymPy with this regard. As I said, I am not a NumPy developer and as such, this decision should be made by NumPy developers and Travis as the original project leader. I could see a familiar pattern here --- Travis has spent enormous time to develop NumPy and to build a community, and later became busy. This is exactly what happened to me with SymPy (when I was back in Prague, I spent months, every evening, many hours with sympy). In fact, Travis once said at some lecture, that opensource is addictive. And not only that, also, if you develop (start) something, it really feels like it's yours. And then when I didn't have time and I knew I am not doing good job with SymPy, it was probably the hardest decision I had to make to pass the
Re: [Numpy-discussion] rewriting NumPy code in C or C++ or similar
Hi Sturla,
On Tue, Mar 8, 2011 at 6:25 AM, Sturla Molden stu...@molden.no wrote:
Den 08.03.2011 05:05, skrev Dan Halbert:
Thanks, that's a good suggestion. I have not written Fortran since 1971,
but it's come a long way. I was a little worried about the row-major vs
column-major issue, but perhaps that can be handled just by remembering
to reverse the subscript order between C and Fortran.
In practice this is not a problem. Most numerical libraries for C assume
Fortran-ordering, even OpenGL assumes Fortran-ordering. People program
MEX files for Matlab in C all the time. Fortran-ordering is assumed in
MEX files too.
In ANSI C, array bounds must be known at compile time, so a Fortran
routine with the interface
subroutine foobar( lda, A )
integer lda
double precision A(lda,*)
end subroutine
will usually be written like
void foobar( int lda, double A[]);
in C, ignoring different calling convention for lda.
Now we would index A(row,col) in Fortran and A[row + col*lda] in C. Is
that too difficult to remember?
In ANSI C the issue actually only arises with small array of arrays
having static shape, or convoluted contructs like pointer to an array
of pointers to arrays. Just avoid those and stay with 1D arrays in C --
do the 1D to 2D mapping in you mind.
In C99 arrays are allowed to have dynamic size, which mean we can do
void foobar( int lda, double *pA )
{
typedef double row_t [lda];
vector_t *A = (vector_t*)((void*)pA[0]);
Here we must index A[k][i] to match A(i,k) in Fortran. I still have not
seen anyone use C99 like this, so I think it is merely theoretical.
Chances are if you know how to do this with C99, you also know how to
get the subscripts right. If you are afraid to forget to reverse the
subscript order between C and Fortran, it just tells me you don't
really know what you are doing when using C, and should probably use
something else.
Why not Cython? It has native support for NumPy arrays.
Personally I prefer Fortran 95, but that is merely a matter of taste.
+1 to all that you wrote about Fortran. I am pretty much switching to
it from C/C++ for all my numerical work, and then I use Cython to call
it from Python, as well as cmake for the build system.
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to compile Fortran using setup.py
Hi Pearu!
On Sat, Mar 12, 2011 at 2:30 AM, Pearu Peterson
pearu.peter...@gmail.com wrote:
On Fri, Mar 11, 2011 at 3:58 AM, Ondrej Certik ond...@certik.cz wrote:
Hi,
I spent about an hour googling and didn't figure this out. Here is my
setup.py:
setup(
name = libqsnake,
cmdclass = {'build_ext': build_ext},
version = 0.1,
packages = [
'qsnake',
'qsnake.calculators',
'qsnake.calculators.tests',
'qsnake.data',
'qsnake.mesh2d',
'qsnake.tests',
],
package_data = {
'qsnake.tests': ['phaml_data/domain.*'],
},
include_dirs=[numpy.get_include()],
ext_modules = [Extension(qsnake.cmesh, [
qsnake/cmesh.pyx,
qsnake/fmesh.f90,
])],
description = Qsnake standard library,
license = BSD,
)
You can specify Fortran code, that you don't want to process with f2py, in
the libraries list
and then use the corresponding library in the extension, for example:
setup(...
libraries = [('foo', dict(sources=['qsnake/fmesh.f90']))],
ext_modules = [Extension(qsnake.cmesh,
sources =
[qsnake/cmesh.pyx],
libraries = ['foo']
)],
...
)
See also scipy/integrate/setup.py that resolves the same issue but just
using the configuration function approach.
Indeed, I just tried it and it works! Thanks. I am now able to compile
fortran code into the extension module.
The only problem is that I am not able to convince Cython to also
parse the .pyx files, it clashes with the numpy's implementation of
build_ext. Which usually is not a problem, as long as one doesn't need
to compile Fortran and .pyx at the same time.
So cmake seems like the only robust option anyway.
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to compile Fortran using setup.py
On Fri, Mar 11, 2011 at 12:24 AM, Dag Sverre Seljebotn
d.s.seljeb...@astro.uio.no wrote:
On 03/11/2011 09:13 AM, Dag Sverre Seljebotn wrote:
On 03/11/2011 07:57 AM, Ondrej Certik wrote:
On Thu, Mar 10, 2011 at 8:25 PM, Robert Kernrobert.k...@gmail.com wrote:
On Thu, Mar 10, 2011 at 19:58, Ondrej Certikond...@certik.cz wrote:
Hi,
I spent about an hour googling and didn't figure this out. Here is my
setup.py:
setup(
name = libqsnake,
cmdclass = {'build_ext': build_ext},
version = 0.1,
packages = [
'qsnake',
'qsnake.calculators',
'qsnake.calculators.tests',
'qsnake.data',
'qsnake.mesh2d',
'qsnake.tests',
],
package_data = {
'qsnake.tests': ['phaml_data/domain.*'],
},
include_dirs=[numpy.get_include()],
ext_modules = [Extension(qsnake.cmesh, [
qsnake/cmesh.pyx,
qsnake/fmesh.f90,
])],
description = Qsnake standard library,
license = BSD,
)
The qsnake.cmesh extension needs to compile .pyx into .c and later to
.o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then
link both things. That's it. In other words, this is what I want
distutils to do:
$ cython cmesh.pyx
$ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6
-I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include
$ gfortran -fPIC -o fmesh.o -c fmesh.f90
$ gcc -shared -o cmesh.so cmesh.o fmesh.o
Difficult if sticking with distutils of any flavor. You would have
probably to replace the build_ext command to handle both the Fortran
and the Cython. You may want to give David Cournapeau's Bento a try:
http://pypi.python.org/pypi/bento/
Thanks Robert. I burned most of my today's evening on this, trying to
replace the command, but so far I didn't figure it out. It is indeed
difficult, but I wasn't sure, whether it is because I am not so
familiar with distutils.
Kurt Smith spent much more than an evening on Fortran+distutils for the
first release of Fwrap, I believe you could find something in the Fwrap
0.1 release...
BUT, for Fwrap trunk, Kurt ditched all of that in favour of waf. And it
works much, much better (I can simply export FC=ifort and it finds the
right flags for Intel Fortran). I guess in your case, that means
sticking with cmake.
I looked at bento, but I'll simply stick to cmake. I thought that for
a Python package (that uses cython + C or fortran, which I thought is
a pretty standard configuration for scientific computing), I would use
distutils, just like other Python packages.
The point of Bento as I understand it is exactly to seperate packaging
from building, so that it becomes possible to ditch distutils but still
have things behave like a Python package more or less.
In theory at least you should be able to plug cmake into Bento (perhaps
by having Bento generate files included into your cmake scripts, and
then launching a cmake process).
David is currently working on using Bento and waf together for building
NumPy.
With cmake, I have already figured out how to mix fotran, c, c++,
Cython and Python and everything works nicely together, very robustly.
So if I have to hack distutils anyway, I would write a simple
distutils - cmake bridge.
Please, write a bento+cmake bridge instead :-) Too much time that could
have been spent better has gone into distutils already, it's time to
just stop using it.
Also, bento comes with a distutils bridge now. So with bento-cmake, one
should be able to go distutils-bento-cmake.
Thanks Dag for the explanation about bento. Indeed, you are right,
what I want is a solid build system, that can do the job (let's say
cmake in my case), and also create Pythonic packaging, so that it does
what people expect from a Python package.
So the right approach in my case is to stick to distutils to support
setup.py, use bento to hook things in, and write a simple bento-cmake
bridge if I have time.
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to compile Fortran using setup.py
Hi,
I spent about an hour googling and didn't figure this out. Here is my setup.py:
setup(
name = libqsnake,
cmdclass = {'build_ext': build_ext},
version = 0.1,
packages = [
'qsnake',
'qsnake.calculators',
'qsnake.calculators.tests',
'qsnake.data',
'qsnake.mesh2d',
'qsnake.tests',
],
package_data = {
'qsnake.tests': ['phaml_data/domain.*'],
},
include_dirs=[numpy.get_include()],
ext_modules = [Extension(qsnake.cmesh, [
qsnake/cmesh.pyx,
qsnake/fmesh.f90,
])],
description = Qsnake standard library,
license = BSD,
)
The qsnake.cmesh extension needs to compile .pyx into .c and later to
.o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then
link both things. That's it. In other words, this is what I want
distutils to do:
$ cython cmesh.pyx
$ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6
-I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include
$ gfortran -fPIC -o fmesh.o -c fmesh.f90
$ gcc -shared -o cmesh.so cmesh.o fmesh.o
If I import the relevant commands above by:
from numpy.distutils.core import setup
from numpy.distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy
then it gives me:
running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands
--compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands
--fcompiler options
running build_src
build_src
building extension qsnake.cmesh sources
f2py options: []
f2py: build/src.linux-x86_64-2.6/qsnake/cmeshmodule.c
creating build
creating build/src.linux-x86_64-2.6
creating build/src.linux-x86_64-2.6/qsnake
Reading fortran codes...
Reading file 'qsnake/fmesh.f90' (format:free)
Traceback (most recent call last):
File setup.py, line 29, in module
license = BSD,
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/core.py,
line 186, in setup
return old_setup(**new_attr)
File /home/certik1/repos/qsnake/local/lib/python/distutils/core.py,
line 152, in setup
dist.run_commands()
File /home/certik1/repos/qsnake/local/lib/python/distutils/dist.py,
line 975, in run_commands
self.run_command(cmd)
File /home/certik1/repos/qsnake/local/lib/python/distutils/dist.py,
line 995, in run_command
cmd_obj.run()
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/install.py,
line 55, in run
r = old_install.run(self)
File
/home/certik1/repos/qsnake/local/lib/python/distutils/command/install.py,
line 577, in run
self.run_command('build')
File /home/certik1/repos/qsnake/local/lib/python/distutils/cmd.py,
line 333, in run_command
self.distribution.run_command(command)
File /home/certik1/repos/qsnake/local/lib/python/distutils/dist.py,
line 995, in run_command
cmd_obj.run()
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build.py,
line 37, in run
old_build.run(self)
File /home/certik1/repos/qsnake/local/lib/python/distutils/command/build.py,
line 134, in run
self.run_command(cmd_name)
File /home/certik1/repos/qsnake/local/lib/python/distutils/cmd.py,
line 333, in run_command
self.distribution.run_command(command)
File /home/certik1/repos/qsnake/local/lib/python/distutils/dist.py,
line 995, in run_command
cmd_obj.run()
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py,
line 152, in run
self.build_sources()
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py,
line 169, in build_sources
self.build_extension_sources(ext)
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py,
line 334, in build_extension_sources
sources = self.f2py_sources(sources, ext)
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py,
line 593, in f2py_sources
['-m',ext_name]+f_sources)
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/f2py2e.py,
line 342, in run_main
postlist=callcrackfortran(files,options)
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/f2py2e.py,
line 276, in callcrackfortran
postlist=crackfortran.crackfortran(files)
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py,
line 2683, in crackfortran
readfortrancode(files,crackline)
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py,
line 420, in readfortrancode
dowithline(finalline)
File
/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py,
line 624, in crackline
analyzeline(m,pat[1],line)
File
Re: [Numpy-discussion] how to compile Fortran using setup.py
On Thu, Mar 10, 2011 at 8:25 PM, Robert Kern robert.k...@gmail.com wrote:
On Thu, Mar 10, 2011 at 19:58, Ondrej Certik ond...@certik.cz wrote:
Hi,
I spent about an hour googling and didn't figure this out. Here is my
setup.py:
setup(
name = libqsnake,
cmdclass = {'build_ext': build_ext},
version = 0.1,
packages = [
'qsnake',
'qsnake.calculators',
'qsnake.calculators.tests',
'qsnake.data',
'qsnake.mesh2d',
'qsnake.tests',
],
package_data = {
'qsnake.tests': ['phaml_data/domain.*'],
},
include_dirs=[numpy.get_include()],
ext_modules = [Extension(qsnake.cmesh, [
qsnake/cmesh.pyx,
qsnake/fmesh.f90,
])],
description = Qsnake standard library,
license = BSD,
)
The qsnake.cmesh extension needs to compile .pyx into .c and later to
.o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then
link both things. That's it. In other words, this is what I want
distutils to do:
$ cython cmesh.pyx
$ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6
-I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include
$ gfortran -fPIC -o fmesh.o -c fmesh.f90
$ gcc -shared -o cmesh.so cmesh.o fmesh.o
Difficult if sticking with distutils of any flavor. You would have
probably to replace the build_ext command to handle both the Fortran
and the Cython. You may want to give David Cournapeau's Bento a try:
http://pypi.python.org/pypi/bento/
Thanks Robert. I burned most of my today's evening on this, trying to
replace the command, but so far I didn't figure it out. It is indeed
difficult, but I wasn't sure, whether it is because I am not so
familiar with distutils.
I looked at bento, but I'll simply stick to cmake. I thought that for
a Python package (that uses cython + C or fortran, which I thought is
a pretty standard configuration for scientific computing), I would use
distutils, just like other Python packages.
With cmake, I have already figured out how to mix fotran, c, c++,
Cython and Python and everything works nicely together, very robustly.
So if I have to hack distutils anyway, I would write a simple
distutils - cmake bridge.
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Trac unaware of github move
On Sun, Nov 7, 2010 at 7:17 AM, Ralf Gommers ralf.gomm...@googlemail.com wrote: On Thu, Nov 4, 2010 at 10:06 PM, Pauli Virtanen p...@iki.fi wrote: Thu, 04 Nov 2010 22:00:47 +0800, Ralf Gommers wrote: I just noticed that the Trac wiki is not displaying updates to files kept in the source tree, for example http://projects.scipy.org/numpy/wiki/TestingGuidelines is stuck at an older version. Can one of the admins point the ReST plugin to github? That would require more work on the Trac-Git integration front: http://github.com/pv/githubsimple-trac It might be more cost-effective to just use links to the Github web interface. That will require renaming those files in the source tree from *.txt to *.rst, otherwise there's no way to have github render them properly. Unless I missed something. Would that be fine? I use .rst instead of .txt for my projects, so that github can render it properly. I don't know if that's the right solution, but it gets the job done. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help with the tensordot function?
Hi Rick! On Fri, Sep 3, 2010 at 4:02 AM, Rick Muller rpmul...@gmail.com wrote: Can someone help me replace a slow expression with a faster one based on tensordot? I've read the documentation and I'm still confused. I have two matrices b and d. b is n x m and d is m x m. I want to replace the expression bdb = zeros(n,'d') for i in xrange(n): bdb[i,:] = dot(b[i,:],dot(d,b[i,:]) I am first trying to reproduce this --- the above is missing one ) and also dot() seems to produce a number, but you are assigning it to bdb[i,:], also you declare bdb as a 1D array. So I tried this: http://gist.github.com/568879/ with something that doesn't have the for loop and thus is a bit faster. The first step is bd = dot(b,d) However, following this with bdb = dot(bd,b.T) doesn't work, since it yields a n x n matrix instead of an n x 1 vector. Reading the notes on tensordot makes me think it's the function to use, but I'm having trouble grokking the axes argument. Can anyone help? In the above gist, I did the following: bd = dot(b, d) bdb = diag(dot(bd, b.T)) print bdb which printed the same as: for i in xrange(n): bdb[i] = dot(b[i,:], dot(d, b[i, :])) print bdb but I think that this is not what you want, is it? I think you want to do something like: b * d * b.T but since b is a (n, m) matrix, the result is a matrix, not a vector, right? Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help with the tensordot function?
On Tue, Sep 7, 2010 at 12:23 PM, Ondrej Certik ond...@certik.cz wrote: Hi Rick! On Fri, Sep 3, 2010 at 4:02 AM, Rick Muller rpmul...@gmail.com wrote: Can someone help me replace a slow expression with a faster one based on tensordot? I've read the documentation and I'm still confused. I have two matrices b and d. b is n x m and d is m x m. I want to replace the expression bdb = zeros(n,'d') for i in xrange(n): bdb[i,:] = dot(b[i,:],dot(d,b[i,:]) I am first trying to reproduce this --- the above is missing one ) and also dot() seems to produce a number, but you are assigning it to bdb[i,:], also you declare bdb as a 1D array. So I tried this: http://gist.github.com/568879/ with something that doesn't have the for loop and thus is a bit faster. The first step is bd = dot(b,d) However, following this with bdb = dot(bd,b.T) doesn't work, since it yields a n x n matrix instead of an n x 1 vector. Reading the notes on tensordot makes me think it's the function to use, but I'm having trouble grokking the axes argument. Can anyone help? In the above gist, I did the following: bd = dot(b, d) bdb = diag(dot(bd, b.T)) print bdb which printed the same as: for i in xrange(n): bdb[i] = dot(b[i,:], dot(d, b[i, :])) print bdb but I think that this is not what you want, is it? I think you want to do something like: b * d * b.T but since b is a (n, m) matrix, the result is a matrix, not a vector, right? Ah, I just noticed you got this resolved in the other thread. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] update on the transition to git/github
On Tue, Jul 13, 2010 at 7:50 PM, Joshua Holbrook josh.holbr...@gmail.com wrote: This is awesome! I love github. I really wanted to champion for its use at the BoF but unfortunately missed it. Indeed, that's awesome. I should say finally! :) Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] comparing floating point numbers
Hi, I was always using something like abs(x-y) eps or (abs(x-y) eps).all() but today I needed to also make sure this works for larger numbers, where I need to compare relative errors, so I found this: http://www.cygnus-software.com/papers/comparingfloats/comparingfloats.htm and wrote this: def feq(a, b, max_relative_error=1e-12, max_absolute_error=1e-12): a = float(a) b = float(b) # if the numbers are close enough (absolutely), then they are equal if abs(a-b) max_absolute_error: return True # if not, they can still be equal if their relative error is small if abs(b) abs(a): relative_error = abs((a-b)/b) else: relative_error = abs((a-b)/a) return relative_error = max_relative_error Is there any function in numpy, that implements this? Or maybe even the better, integer based version, as referenced in the link above? I need this in tests, where I calculate something on some mesh, then compare to the correct solution projected on some other mesh, so I have to deal with accuracy issues. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] PyArray_SimpleNewFromData segfaults
Hi,
I am getting a segfault in PyArray_SimpleNewFromData in Cython. I am
trying to debug it for the last 4 hours, but still absolutely no clue,
so I am posting it here, maybe someone knows where the problem is:
cdef ndarray array_double_c2numpy(double *A, int len):
from numpy import empty
print got len:, len
cdef npy_intp dims[10]
cdef double X[500]
print 1
dims[0] = 3
print 2x
print dims[0], len
print X[0], X[1], X[2]
cdef npy_intp size
cdef ndarray newarr
cdef double *arrsource
size = 10
arrsource = double *malloc(sizeof(double) * size)
print still alive
newarr = PyArray_SimpleNewFromData(1, size, 12,
void *arrsource)
print I am already dead. :(
print 3
return empty([len])
Essential is just the line:
newarr = PyArray_SimpleNewFromData(1, size, 12,
void *arrsource)
Then I removed all numpy from my computer, downloaded the latest git
repository from:
http://projects.scipy.org/git/numpy.git
applied the following patch:
diff --git a/numpy/core/src/multiarray/ctors.c b/numpy/core/src/multiarray/ctors
index 3fdded0..777563c 100644
--- a/numpy/core/src/multiarray/ctors.c
+++ b/numpy/core/src/multiarray/ctors.c
@@ -1318,6 +1318,7 @@ PyArray_NewFromDescr(PyTypeObject *subtype, PyArray_Descr
intp *dims, intp *strides, void *data,
int flags, PyObject *obj)
{
+printf(entering PyArray_NewFromDescr\n);
PyArrayObject *self;
int i;
size_t sd;
@@ -1553,6 +1554,7 @@ PyArray_New(PyTypeObject *subtype, int nd, intp *dims, int
{
PyArray_Descr *descr;
PyObject *new;
+printf(entering PyArray_New, still kicking\n);
descr = PyArray_DescrFromType(type_num);
if (descr == NULL) {
then installed with:
python setup.py install --home=~/usr
and run my cython program. Here is the output:
$ ./schroedinger
---
This is Hermes1D - a free ODE solver
based on the hp-FEM and Newton's method,
developed by the hp-FEM group at UNR
and distributed under the BSD license.
For more details visit http://hpfem.org/.
---
Importing hermes1d
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_New, still kicking
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_New, still kicking
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
Python initialized
got len: 39601
1
2x
3 39601
0.0 0.0 0.0
still alive
Segmentation fault
What puzzles me is that there is no debugging print statement just
before the segfault. So like if the PyArray_New was not being called.
But looking into numpy/core/include/numpy/ndarrayobject.h, line 1359:
#define PyArray_SimpleNewFromData(nd, dims, typenum, data)\
PyArray_New(PyArray_Type, nd, dims, typenum, NULL, \
data, 0, NPY_CARRAY, NULL)
It should be called. Does it segfault in the printf() statement above?
Hm. I also tried gdb, but it doesn't step into
PyArray_SimpleNewFromData (in the C file), not sure why.
So both print statements and gdb failed to bring me to the cause,
pretty sad day for me. I am going home now and start with a fresh
head, it just can't segfault like this... I guess I'll start by
creating a simple cython project to reproduce it (the schroedinger
code above is quite involved, it starts a python interpreter inside a
C++ program, etc. etc.).
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_SimpleNewFromData segfaults
On Mon, Oct 5, 2009 at 7:34 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Oct 5, 2009 at 7:40 PM, Ondrej Certik ond...@certik.cz wrote: [...] still alive Segmentation fault What puzzles me is that there is no debugging print statement just before the segfault. Maybe you need to flush the buffer. That is a good thing to do when segfaults are about. I tried to put fflush(NULL); after it, but it didn't help. I have created a super simple demo for anyone to play: $ git clone git://github.com/certik/segfault.git $ cd segfault/ $ vim Makefile # -- edit the python and numpy include paths $ make $ python test.py I am still alive Segmentation fault where test.py is: $ cat test.py import _hermes1d v = _hermes1d.test() print v and _hermes1d.pyx is: $ cat _hermes1d.pyx def test(): cdef npy_intp size cdef ndarray newarr cdef double *arrsource size = 10 arrsource = double *malloc(sizeof(double) * size) print I am still alive newarr = PyArray_SimpleNewFromData(1, size, NPY_DOUBLE, void *arrsource) print I am dead. return newarr So I bet there is something very stupid that I am missing. Still investigating... Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_SimpleNewFromData segfaults
On Mon, Oct 5, 2009 at 8:38 PM, Ondrej Certik ond...@certik.cz wrote: On Mon, Oct 5, 2009 at 7:34 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Oct 5, 2009 at 7:40 PM, Ondrej Certik ond...@certik.cz wrote: [...] still alive Segmentation fault What puzzles me is that there is no debugging print statement just before the segfault. Maybe you need to flush the buffer. That is a good thing to do when segfaults are about. I tried to put fflush(NULL); after it, but it didn't help. I have created a super simple demo for anyone to play: $ git clone git://github.com/certik/segfault.git $ cd segfault/ $ vim Makefile # -- edit the python and numpy include paths $ make $ python test.py I am still alive Segmentation fault where test.py is: $ cat test.py import _hermes1d v = _hermes1d.test() print v and _hermes1d.pyx is: $ cat _hermes1d.pyx def test(): cdef npy_intp size cdef ndarray newarr cdef double *arrsource size = 10 arrsource = double *malloc(sizeof(double) * size) print I am still alive newarr = PyArray_SimpleNewFromData(1, size, NPY_DOUBLE, void *arrsource) print I am dead. return newarr So I bet there is something very stupid that I am missing. Still investigating... I didn't call _import_array() ! This patch fixes it: diff --git a/_hermes1d.pxd b/_hermes1d.pxd index 9994c28..f5e8868 100644 --- a/_hermes1d.pxd +++ b/_hermes1d.pxd @@ -54,6 +54,8 @@ cdef extern from arrayobject.h: object PyArray_SimpleNewFromData(int nd, npy_intp* dims, int typenum, void* data) +void _import_array() + cdef extern from Python.h: ctypedef void PyObject void Py_INCREF(PyObject *x) diff --git a/_hermes1d.pyx b/_hermes1d.pyx index e542ddc..7a4beec 100644 --- a/_hermes1d.pyx +++ b/_hermes1d.pyx @@ -2,6 +2,7 @@ def test(): cdef npy_intp size cdef ndarray newarr cdef double *arrsource +_import_array() size = 10 arrsource = double *malloc(sizeof(double) * size) I think I learned something today the hard way. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_SimpleNewFromData segfaults
On Mon, Oct 5, 2009 at 9:25 PM, Ondrej Certik ond...@certik.cz wrote: On Mon, Oct 5, 2009 at 8:38 PM, Ondrej Certik ond...@certik.cz wrote: On Mon, Oct 5, 2009 at 7:34 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Oct 5, 2009 at 7:40 PM, Ondrej Certik ond...@certik.cz wrote: [...] still alive Segmentation fault What puzzles me is that there is no debugging print statement just before the segfault. Maybe you need to flush the buffer. That is a good thing to do when segfaults are about. I tried to put fflush(NULL); after it, but it didn't help. I have created a super simple demo for anyone to play: $ git clone git://github.com/certik/segfault.git $ cd segfault/ $ vim Makefile # -- edit the python and numpy include paths $ make $ python test.py I am still alive Segmentation fault where test.py is: $ cat test.py import _hermes1d v = _hermes1d.test() print v and _hermes1d.pyx is: $ cat _hermes1d.pyx def test(): cdef npy_intp size cdef ndarray newarr cdef double *arrsource size = 10 arrsource = double *malloc(sizeof(double) * size) print I am still alive newarr = PyArray_SimpleNewFromData(1, size, NPY_DOUBLE, void *arrsource) print I am dead. return newarr So I bet there is something very stupid that I am missing. Still investigating... I didn't call _import_array() ! This patch fixes it: diff --git a/_hermes1d.pxd b/_hermes1d.pxd index 9994c28..f5e8868 100644 --- a/_hermes1d.pxd +++ b/_hermes1d.pxd @@ -54,6 +54,8 @@ cdef extern from arrayobject.h: object PyArray_SimpleNewFromData(int nd, npy_intp* dims, int typenum, void* data) + void _import_array() + cdef extern from Python.h: ctypedef void PyObject void Py_INCREF(PyObject *x) diff --git a/_hermes1d.pyx b/_hermes1d.pyx index e542ddc..7a4beec 100644 --- a/_hermes1d.pyx +++ b/_hermes1d.pyx @@ -2,6 +2,7 @@ def test(): cdef npy_intp size cdef ndarray newarr cdef double *arrsource + _import_array() size = 10 arrsource = double *malloc(sizeof(double) * size) I think I learned something today the hard way. The only mention of the _import_array() in the documentation that I found is here: http://docs.scipy.org/doc/numpy/reference/c-api.array.html#NO_IMPORT_ARRAY but I don't understand what it means do I have to just call _import_array() and then I can use numpy CAPI, or do I also have to define those PY_ARRAY_UNIQUE_SYMBOL etc? Btw, to explain my original post for future readers --- the real problem was that PyArray_Type was NULL and thus PyArray_Type segfaulted. That happened in the definition: #define PyArray_SimpleNewFromData(nd, dims, typenum, data)\ PyArray_New(PyArray_Type, nd, dims, typenum, NULL, \ data, 0, NPY_CARRAY, NULL) so it is *extremely* confusing, since PyArray_SimpleNewFromData() was being called from my code, but PyArray_New never started, it segfaulted in between. I think now it is clear what is going on. Only I don't understand the intention, but I can now get my job done. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
2009/4/13 Stéfan van der Walt ste...@sun.ac.za: 2009/4/13 Eric Firing efir...@hawaii.edu: Stéfan, or other git-users, One feature of hg that I use frequently is hg serve, the builtin web server. I use it for two purposes: for temporary local publishing (e.g., in place of using ssh--sometimes it is quicker and easier), and for providing access to the very nice hgwebdir.cgi browsing capability on local repos. I have looked through git commands etc., and have not found an equivalent; am I missing something? The browsing capability of hgwebdir.cgi is much better than any gui interface I have seen for git or for hg. I realize there is a gitweb.cgi, but having that cgi is not the same as being able to publish locally with a single command, and then browse. The command is git instaweb --httpd=webrick Ah, that's nice, thanks for sharing it. I didn't know about it. Works fine for me. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Sun, Apr 12, 2009 at 7:13 PM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Eric Firing wrote: It probably does--it is written in python. Yes, but it is just a script to call git-daemon. I am somewhat doubtful that it would work on windows, but I would love being proven wrong :) It uses os.fork() which doesn't work on windows. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Sat, Apr 11, 2009 at 11:44 AM, Neal Becker ndbeck...@gmail.com wrote: Ondrej Certik wrote: On Fri, Apr 10, 2009 at 12:43 AM, Eric Firing efir...@hawaii.edu wrote: Ondrej Certik wrote: On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Ondrej Certik wrote: It is maybe easier to learn how to work with different clones, but once you start working with lots of patches and you need to reclone all the time, then it's the wrong approach to work, as it takes lots of time to copy the whole repository on the disk. This is simply wrong. Mercurial uses hard links for cloning a repo that On my laptop, recloning the whole repository (with hardlinks) takes considerably longer than creating a new branch in the same directory, that's a pure fact. You can clone a repo using: cp -al old new That should be very fast. As long as you then use an editor that behaves correctly with hard links. If you use emacs you can configure this behavior. Ok, this seems to be pretty fast: $ time cp -al sympy-git.hg/ new real0m0.129s user0m0.020s sys 0m0.084s e.g. this was the mercurial repo. Creating a new branch in git: $ time git co -b new2 Switched to a new branch new real0m0.048s user0m0.020s sys 0m0.016s is faster, but I agree, that 0.1s is not an issue for me. Is this going to work on windows? I thought windows don't have hardlinks. In any case, I would prefer to use standard mercurial tools for the job, so if I do: $ time hg clone sympy-git.hg new updating working directory 566 files updated, 0 files merged, 0 files removed, 0 files unresolved real0m1.156s user0m0.948s sys 0m0.164s it still takes over a second. That's too much for me, as this operation of creating new branches is something that I do almost all the time. The way out is to use branches in on repository, and imho that's the correct way, both in git and in mercurial. However, is here anyone who actually uses branches in mercurial? If so, try this: hg clone http://hg.sympy.org/sympy-git.hg cd sympy-git.hg hg branch new2 vim README # do some changes hg ci hg up -C default hg vi and the hg vi doesn't even show your branch names and which branch contains what. let's edit README in the default branch: vim README hg ci now if you do: hg vi it shows the new2 branch, and it shows the main branch diverged, so it doesn't look as nice as in gitk, but it is possible to use. Now let's try mercurial rebase: $ hg up -C new2 $ hg rebase -d default merging README saving bundle to /tmp/ab/hg/sympy-git.hg/.hg/strip-backup/536215160a1c-temp adding branch adding changesets adding manifests adding file changes added 2 changesets with 2 changes to 1 files abort: 00changelo...@536215160a1c: no node! Oops, something went wrong. But commits are still there, so I guess I can safely ignore the error message (could someone clarify?). Now let's say I would like to merge the top two patches, since they both modify readme and I would like to only send one patch. In git, I just do git rebase -i HEAD~2 tell it in vim which patches to squash and I am done. In mercurial, it's a hell of a job: $ hg qimport -r tip $ hg qimport -r qparent $ hg qpop now at: 2146.diff $ hg qfold 2147.diff $ hg qdelete -r qbase:qtip And I am still not done! I forgot to change the log (git asks you this automatically during the rebase), so we need to import the patch to MQ again: $ hg qimport -r tip $ hg qrefresh -e $ hg qdelete -r qbase:qtip And I am finally done. Now let's say some of the patches in MQ didn't apply after changing the order or some other changes. Then I am in deep troubles, because hg qpush fails and I need to modify the patch by hand (that really sucks!). With git, you only use rebase, and rebase is pretty powerful tool that can handle most of the conflicts itself, and if it can't, it asks you to resolve it, I assume just like mercurial rebase, but unfortunately mercurial rebase can't be used to mangle patches or history. I would like to ask mercurial proponents on this list to please correct me above and do it the right way. :) So that it's not biased. Also, read this nice article, that imho nicely compares git and mercurial: http://rg03.wordpress.com/2009/04/07/mercurial-vs-git/ Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Sat, Apr 11, 2009 at 2:12 PM, Neal Becker ndbeck...@gmail.com wrote: Why not try asking on mercur...@selenic.com (gmane.comp.version- control.mercurial.general) Done: http://www.selenic.com/pipermail/mercurial/2009-April/025131.html O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Sat, Apr 11, 2009 at 8:28 PM, Fernando Perez fperez@gmail.com wrote: In bzr (and as far as I see, also in hg), this kind of history rewriting is near impossible, so the best you can do is make a merge commit and leave all that history in there, visible in the 'second level' log (indented in the text view). As a project grows many developers, having all this history merged back into the main project tree gets unwieldy. Great email. Exactly, that's my experience with hg, it makes it nearly impossible, without misusing mercurial queues. I am just surprised, you have exactly the same experience with bzr, I thought it's only hg. As an amusing story, my roommate here at UNR also switched from svn to hg and was telling me, you know, git is too unfriendly, hg does everything I ever need. I was telling myself, well, after you really start using it, you'll maybe switch to git. Then after couple months he once told me -- I had to rewrite history, so I tried MQ and that's a disaster, the interface to it is horrible. So I switched to git. I was thinking about Fernando just yesterday, asking myself, hm, I wonder how he is satisfied with bzr. So this email really made me laugh. :) As for what numpy/scipy should do, I'll leave that to those who actually contribute to the projects :) I just hope that this view is Exactly, this decision should be done by numpy developers, not me either. useful to anyone who wants to think about the problem from an angle different to that of specific commands, benchmarks or features :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Ondrej Certik wrote: It is maybe easier to learn how to work with different clones, but once you start working with lots of patches and you need to reclone all the time, then it's the wrong approach to work, as it takes lots of time to copy the whole repository on the disk. Yes, *I* know how to use git, and I agree with you, I vastly prefer git branch handling to bzr branch handling. *I* find working with GUI for VCS a real PITA. But I am not the only numpy developer, that's why the feedback from people like Josef with a totally different workflow than me is valuable - much more than people like us who are unix geeks :) Yes, definitely. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Fri, Apr 10, 2009 at 1:07 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Eric Firing wrote: This is simply wrong. Mercurial uses hard links for cloning a repo that is on the same disk, so it is faster and much more space-efficient than copying the files. Yes, but maybe Ondrej talks about an older hg version ? Hg could not handle NTFS hardlink for some time, but it does now if you have pywin32. And still, switching between branches is faster in git than in hg, for some technical reasons I can't really explain (but can be found on the ML archives). But as I said previously, speed is not really an argument for me. If hg is fast enough for python, it is obviously fast enough for numpy and scipy. As long as it does not takes minutes to merge/review the 5 lines difference between two branches as is the case in svn right now, I am happier :) But if you do want named branches in a given repo, you can have that also with hg. Granted, it has not always been part of hg, but it is now. Same with rebasing and transplanting. As I understand, and correct me if I am wrong, the problems with named branches are: - you can't remove them later, it is in the repository 'forever' - it is not easy to make them publicly available Plus with git, you can fetch the remote repository with all the branches and browse them locally in your remote branches, when you are offline. And merge them with your own branches. In mercurial, it seems the only way to see what changes are there and which branch and which commits I want to merge is to use hg in, but that requires the internet connection, so it's basically like svn. I don't know if mercurial has improved in this lately, but at least for me, that's a major problem with mercurial. But since python now moved to mercurial too, maybe they will help fix this. :) It seems to me like the python distutils vs cmake discussion. I also prefer the unpythonic cmake. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
* I use Python for a bunch of other stuff Matlab is not suitable for -- This is my argument about usability and tool support. A few years back, CVS was a standard, now SVN is. I like that I can use the same tool to contribute to a whole bunch of OS projects, and I use it to manage all my projects as work. It seems many in the OS world are moving to a DVCS -- but there are three major ones in play: git, Hg and bzr -- I don't know enough about any of them to say what I prefer, but I really don't want to have to learn all three! And if I do, it would be nice if there were semi-similar interfaces for them all: tortoise, for instance. Unfortunately, one has to learn all 3 (I need to use bzr for ipython, hg for Sage, maybe soon some other projects and git for everything else). But it's not difficult. I think it's like learning bash and then couple commands from tcsh and zsh just to be able to survive. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Wed, Apr 8, 2009 at 11:04 PM, David Cournapeau courn...@gmail.com wrote: On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik ond...@certik.cz wrote: Yes. Do you have any windows developers (I am sorry, I am not familiar at all with sympy)? Not much. My main concern with git are: - you may need the command line Oh, definitely. What else? :) - the index can be confusing (you can avoid learning it at first, but still some errors won't make sense before you get it). Yes, but in fact the staging area (if this is what you mean) is in every VCS, only it's hidden, except git, where it is made explicit. - git is not discoverable (you need to read some documentation) Yes, but git help command is very detailed most of the time. I tend to think those problems are not that big, and the concern raised on python ML way overblown - but I learned the tool, and I don't think anyone wants to alienate any numpy/scipy developer. Which tool did you learn? Git? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Thu, Apr 9, 2009 at 3:04 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Ondrej Certik wrote: Yes, but in fact the staging area (if this is what you mean) is in every VCS, only it's hidden, except git, where it is made explicit. I am not sure the staging area concept is there in other vcs, because in git it is intrinsically linked to idea that git tracks content. It is If you do hg add, you add to the staging area. If you do hg commit, you commit it, together with automatically adding all changes to the staging area as well (and committing). I think it's the same with svn. powerful, and I really miss it in other DVCS, but it takes time to get used to - time that other people may not be willing to spend. I tried to write some basic instructions which totally bypass the index concept, but it can still bite you even in those simple cases: http://projects.scipy.org/numpy/wiki/GitWorkflow#Commonscenario Which tool did you learn? Git? git. Both Stefan and me really like git, but assuming we make the transition, we worry a bit about the transition cost for people who are happy with svn. I think it won't. At least it didn't in the sympy case. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Thu, Apr 9, 2009 at 7:15 AM, Matthieu Brucher matthieu.bruc...@gmail.com wrote: 2009/4/9 David Cournapeau courn...@gmail.com: On Thu, Apr 9, 2009 at 10:38 PM, Matthieu Brucher matthieu.bruc...@gmail.com wrote: 2009/4/9 David Cournapeau courn...@gmail.com: On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik ond...@certik.cz wrote: Yes. Do you have any windows developers (I am sorry, I am not familiar at all with sympy)? My main concern with git are: - you may need the command line - the index can be confusing (you can avoid learning it at first, but still some errors won't make sense before you get it). - git is not discoverable (you need to read some documentation) I tried to install git on my computer, as git-svn seemed so nice. I tried git-svn and it told me that some .pm files are missing. Why did it install git-svn if some files are missing? And why isn't it possible to get some information on how to install those missing files on the Internet. For which system did you install it ? I don't think git-svn work well or at all on windows. Otherwise, git has binary installers for every platform which matters. Installing from sources is a bit harder than bzr/hg, but certainly easier than svn, if you need to, Installed from source on a RH4. It complains about missing SVN/Core.pm. One thing about git-svn is that this is not really needed if you just use git and I installed git from source on many linuxes and clusters and it just works, as it is just pure C. I usually just use git-svn on my laptop/workstation, where I install the Debian/Ubuntu packages, and I create the git repository, upload to github.com or somewhere else and just work with the git repository. But I agree that if it installs git-svn and it doesn't just work, it's a big problem. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Thu, Apr 9, 2009 at 11:16 AM, Andrew Straw straw...@astraw.com wrote: Matthieu Brucher wrote: One thing about git-svn is that this is not really needed if you just use git and I installed git from source on many linuxes and clusters and it just works, as it is just pure C. I usually just use git-svn on my laptop/workstation, where I install the Debian/Ubuntu packages, and I create the git repository, upload to github.com or somewhere else and just work with the git repository. But I agree that if it installs git-svn and it doesn't just work, it's a big problem. I was inquiring the use of git with the use of one of our internal svn repositories, just to have a feeling about it :( My opinion is that attempting to use git-svn to get a feeling of git is not a good idea. There's too much slowness of svn involved, too much pain of trying to learn git while also trying to learn git-svn (which itself has corner cases and such that pure git doesn't) and there's no bidirectional 1:1 mapping between branches (that I've found), eliminating a huge part of the joy of git -- cheap branches. Better to start developing on a pure git project to get a feel. You can't go wrong with sympy, for example. :) Oh, definitely. :) Here is a list of easy to fix issues: http://code.google.com/p/sympy/issues/list?q=label:EasyToFix if you want to learn it on some easy, but real world example. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Thu, Apr 9, 2009 at 10:25 PM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: josef.p...@gmail.com wrote: So, for my style, working with different clones instead of branches seems easier. Yes, it is. There is no denying that git makes it more difficult for this workflow, and that git is more difficult at first for easy tasks. I am interested in making it as easy as possible, and if it is indeed too difficult, then maybe git is out. It is maybe easier to learn how to work with different clones, but once you start working with lots of patches and you need to reclone all the time, then it's the wrong approach to work, as it takes lots of time to copy the whole repository on the disk. I worked like that with mercurial and I hated it, especially once I started to have ~15 branches. Git branches are cheap and fast. Besides, imho, you can work with different clones with git too, if you want. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
2009/4/7 Stéfan van der Walt ste...@sun.ac.za: 2009/4/6 Ondrej Certik ond...@certik.cz: FWIW, I tend to agree that Hg is less disruptive than git when coming from svn, at least for the simple tasks (I don't know hg enough to have a really informed opinion for more advanced workflows). Yes, git rocks. Did Sympy move from Hg to Git? Yes. If so, could you give us an overview of the pros and cons you experienced? Here are some my (subjective) observations: 1) I think all developers were able to learn git pretty easily. See also: http://wiki.sympy.org/wiki/Git_hg_rosetta_stone 2) git is faster 3) it boosted my productivity a lot, thanks to branches. Previously I was copying the whole directories with sympy, everytime someone sended a patch, just to try it. I know that hg has branches too, but imho working with them is a pain, as you can't rebase patches easily, no interactive rebase etc and also branches were in git from the beginning, so it's very polished. Hg is catching up I guess, but some things are still missing. 4) no need to learn mercurial queues, just learn git and git rebase -i, and that's all I ever need (and it's actually more powerful than MQ). So git is simpler. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Mon, Mar 30, 2009 at 10:59 PM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Eric Firing wrote: I agree. The PEP does not show overwhelming superiority (or, arguably, even mild superiority) of any alternative I think this PEP was poorly written. You can't see any of the advantage/differences of the different systems. Some people even said they don't see the differences with svn. I think the reason partly is that the PEP focused on existing python workflows, but the whole point, at least for me, is to change the general workflow (for reviews, code contributions, etc...). Stephen J. Turnbull sums it up nicely: http://mail.python.org/pipermail/python-dev/2009-March/087968.html FWIW, I tend to agree that Hg is less disruptive than git when coming from svn, at least for the simple tasks (I don't know hg enough to have a really informed opinion for more advanced workflows). Yes, git rocks. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem looping over numpy array in C
On Fri, Feb 20, 2009 at 12:19 PM, Sumant S.R. Oemrawsingh
soemr...@xs4all.nl wrote:
Hi guys,
I have a problem with looping over numpy arrays in C. I modify the array
in-place (and return None), but after modification, the array doesn't seem
to play nice any more.
Below, I have the C code for a function do_something (a stripped version
of my original function), which has as arguments a 1D numpy array (float
or complex) and either an int or a sequence of ints.
In python, I do the following using only the integer:
a = array([0., 1., 2., 3., 2., 1., 0.])
do_something(a,3)
savetxt('bla',a)
Which works fine. However, when I do the same, but with a list of any
length larger than 0:
a = array([0., 1., 2., 3., 2., 1., 0.])
do_something(a,[3])
savetxt('bla',a)
Traceback (most recent call last):
File stdin, line 1, in module
File /usr/lib/python2.5/site-packages/numpy/core/numeric.py, line 767,
in savetxt
X = asarray(X)
File /usr/lib/python2.5/site-packages/numpy/core/numeric.py, line 132,
in asarray
return array(a, dtype, copy=False, order=order)
TypeError: an integer is required
savetxt('bla',a)
For some reason, the first time it doesn't work; the asarray fails but
then succeeds on a second try. The resulting file is identical to when I
would have looped over the list in python, and called the do_something
function with each integer separately. For instance:
for i in [3,1,0]:
do_something(a, i)
works fine as well. So apparently looping in C temporarily leaves the
array in a weird state but manages to automatically be restored after one
exception. I've checked that type(a), a.dtype, a.shape and a.ndim remain
the same before and after calling do_something with a sequence or with an
integer as second argument. That doesn't seem to be the problem.
The reason that I want do the loop in C is that I need some
precalculations, before being able to do the actual loop. But they might
not be time-consuming enough to warrant writing it in C, so I could just
do the loop in python and not have this problem any more. However, if the
loop in C manages to somehow (temporarily) corrupt the array, how can I be
sure that the single-shot call doesn't succeed by accident?
If anyone can help, suggest something to try or spot my mistake, I'd
appreciate it.
I don't have time to go through your code, but I suggest you try
Cython, that plays very nice with numpy.
Ondrej
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] denormal test: what is it for ?
On Tue, Feb 17, 2009 at 8:08 AM, David Cournapeau courn...@gmail.com wrote: Hi, I would like to continue cleaning the setup.py from numpy/core, and there is one test that I can't make sense of: the denormal thing (testcode_mathlib function). The svn log has no information on it (its presence goes back to the first revision of the file). What is it useful for ? Denormal support ? Whether exp works ? I guess noone knows. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy 1.2.1 uploaded to unstable
Hi, I finally found time to upload the numpy 1.2.1 to Debian unstable (currently it's in incoming: http://incoming.debian.org/). The package is lintian clean, but there is one test that failed for me in chroot. I'll wait until it gets to mirrors and then try it on my laptop and report a bug (I uploaded from a ubuntu machine, but of course I compiled it in pbuilder with sid and tested in chroot). Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] parallel compilation of numpy
Hi, I have a shiny new computer with 8 cores and numpy still takes forever to compile --- is there a way to compile it in parallel (make -j9)? Do distutils allow that? If not, let's move to some build system that allows that? Just wanted to check if there is some reason for that, apart from patches welcome. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: HDF5 for Python 1.1
On Mon, Feb 9, 2009 at 12:15 PM, Andrew Collette h...@alfven.org wrote: = Announcing HDF5 for Python (h5py) 1.1 = What is h5py? - HDF5 for Python (h5py) is a general-purpose Python interface to the Hierarchical Data Format library, version 5. HDF5 is a versatile, mature scientific software library designed for the fast, flexible storage of enormous amounts of data. Interesting project. My first question was how it is related to pytables. The answer is here: http://code.google.com/p/h5py/wiki/FAQ#What%27s_the_difference_between_h5py_and_Pytables? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 7:56 PM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Ondrej Certik wrote: That's exactly what I don't like about cmake - it means you can't produce accurate builds (you need to rerun cmake everytime you change the configuration or dependencies, whereas this is automatic with scons/waf). It also have (used to have) very poor documentation unless you buy the book - but it looks like this is changing. You can always rerun cmake if you want (and make it automatic). Imho that's not a problem. But maybe it is done in a better way with scons. I think it is a problem - it means you have to update it explicitly when the configuration changes. In scons, the signature concept is quite powerful: not only are file dependencies handled, but also command lines, etc... For example, in numscons, if you change the blas/lapack from atlas to say MKL, only linking and eventually configuration changes are rebuilt. If you change the fortran compiler but not the C compiler, only fortran code is rebuilt. All of this is 100 % automatic. I got an email from Alexander Neundorf from kde, and with his permission, I am forwarding it here, because he is not subscribed. As he writes below, cmake is in fact cabable of the above. -- Forwarded message -- From: Alexander Neundorf Date: Mon, Feb 9, 2009 at 1:28 PM Subject: cmake information To: Ondrej Certik Hi Ondrej, I hope you are the Ondrej from this thread: http://groups.google.com/group/Numpy-discussion/browse_thread/thread/c12f96b9ac367f57 If not, please let me know and I'll try to find the right email address. I saw this in the thread: --8-88 Ondrej Certik wrote: That's exactly what I don't like about cmake - it means you can't produce accurate builds (you need to rerun cmake everytime you change the configuration or dependencies, whereas this is automatic with scons/waf). It also have (used to have) very poor documentation unless you buy the book - but it looks like this is changing. You can always rerun cmake if you want (and make it automatic). Imho that's not a problem. But maybe it is done in a better way with scons. I think it is a problem - it means you have to update it explicitly when the configuration changes. In scons, the signature concept is quite powerful: not only are file dependencies handled, but also command lines, etc... For example, in numscons, if you change the blas/lapack from atlas to say MKL, only linking and eventually configuration changes are rebuilt. If you change the fortran compiler but not the C compiler, only fortran code is rebuilt. All of this is 100 % automatic. --8-88 CMake does handle this automatically. E.g. if include directories are changed (which you do by editing a CMakeLists.txt or the cmake cache), all files which are affected by the are rebuilt. If some library changes, everything linking to this library is linked again. If any of the files the build depends on (e.g. a CMakeLists.txt, or an included .cmake file, or the cmake cache) is changed, cmake is automatically rerun, and the makefiles/project files are regenerated. I don't know what happens if you are using both C and Fortran in one project and change one of them. I think this is actually not possible, you can (or should) not change the compiler of an existing build tree. Basically everything which uses this compiler is invalid then, the object files, the results of tests etc. I'm not sure handling this separately for different languages within one project is supported by cmake. Alex ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Mon, Feb 9, 2009 at 2:35 PM, Brian Granger ellisonbg@gmail.com wrote: CMake does handle this automatically. E.g. if include directories are changed (which you do by editing a CMakeLists.txt or the cmake cache), all files which are affected by the are rebuilt. If some library changes, everything linking to this library is linked again. If any of the files the build depends on (e.g. a CMakeLists.txt, or an included .cmake file, or the cmake cache) is changed, cmake is automatically rerun, and the makefiles/project files are regenerated. I don't know what happens if you are using both C and Fortran in one project and change one of them. I think this is actually not possible, you can (or should) not change the compiler of an existing build tree. Basically everything which uses this compiler is invalid then, the object files, the results of tests etc. I'm not sure handling this separately for different languages within one project is supported by cmake. Is all of this handled just by calling make (after the initial cmake call), or do you have to first recall cmake and then make? I just tried that and it is handled automatically by calling make. It's really cool! Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 3:10 AM, David Cournapeau courn...@gmail.com wrote: On Sun, Feb 8, 2009 at 3:21 AM, Ondrej Certik ond...@certik.cz wrote: Hi David, Sorry for the confusion: numscons is NOT the preferred build system. The current numpy.distutils extensions, as shipped by numpy, is the preferred one. Numscons is more an experiment, if you want. Ah, I see, thanks for the clarification. So is it supposed to be in Debian? No, I don't think it should be. It is not yet stabilized code wise, so it does not make much sense to package it. Ok. Is numscons supposed to be a build system for other projects as well? Why not to just send the needed patches to scons and just use scons? Because you cannot just use scons. Numscons is a library build on top of scons, for the needs of numpy. There also needs to be some hook from numpy.distutils to use scons (numscons adds a new distutils command, which is used instead of build to build any compiled code-based extensions). Most of the changes needed for scons have been integrated upstream, though, except one or two things. I see. I think it's a bit confusing that one needs to build a new build system just to build numpy, e.g. that both distutils and scons are not good enough. I don't find it that surprising - numpy and scipy require some relatively advanced features (mixed language and cross-platform with support for many toolchains). Within the open source tools, I know only two which can handle those requirements: scons and cmake. For example, it would almost impossible to build numpy/scipy with autoconf. Yes, I am investigating cmake, it's pretty cool. I wrote some macros for cython etc. What I like about cmake is that it is cross platform and it just produces makefiles on linux, or visual studio files (or whatever) on windows. When I get more experience with it, I'll post here. What I don't like on cmake is that it uses it's own language, instead of python, on the other hand, so far everything seems to just work. Contrary to numscons, where it looks almost like a new python program just to build numpy. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 12:26 PM, Brian Granger ellisonbg@gmail.com wrote: Yes, I am investigating cmake, it's pretty cool. I wrote some macros for cython etc. What I like about cmake is that it is cross platform and it just produces makefiles on linux, or visual studio files (or whatever) on windows. When I get more experience with it, I'll post here. Yes, while I haven't played with cmake much, it does look very nice. What I don't like on cmake is that it uses it's own language, instead of python, on the other hand, so far everything seems to just work. I too don't like the idea of cmake having its own language. While I love to learn new languages, I like to have good reasons and I'm not sure that building software is a good reason. I have been playing with Scons and I really do like the fact that it is Python. While I haven't tried it yet, I am pretty sure that we could use IPython to interactively debug a Scons build. That would be really nice! Contrary to numscons, where it looks almost like a new python program just to build numpy. But hold on, it is not fair to compare cmake with numscons (this is an apples and oranges thing). You should compare cmake with *Scons* itself. The problem with comparing cmake with numscons is that cmake can't do what numscons does - i.e., plug into distutils. There is a lot of extra things that distutils does other than build extensions and cmake won't do these things out of the box. Obviously, someone could get cmake to do these things, but then you are developing a complete distutils replacement. And I think that any distutils replacement should done in Python. Yes, I agree with what you said. I still want to try cmake though, if nothing, at least to know where we want to go. Yes, I should compare cmake with scons. We can either develop python front end to cmake, or just help improve scons/numscons etc. I really like that cmake generates native makefiles and windows visual studio files etc. In any case, whatever we choose to use, I want to have experience with both scons and cmake. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
Hi David, Sorry for the confusion: numscons is NOT the preferred build system. The current numpy.distutils extensions, as shipped by numpy, is the preferred one. Numscons is more an experiment, if you want. Ah, I see, thanks for the clarification. So is it supposed to be in Debian? No, I don't think it should be. It is not yet stabilized code wise, so it does not make much sense to package it. Ok. Is numscons supposed to be a build system for other projects as well? Why not to just send the needed patches to scons and just use scons? Because you cannot just use scons. Numscons is a library build on top of scons, for the needs of numpy. There also needs to be some hook from numpy.distutils to use scons (numscons adds a new distutils command, which is used instead of build to build any compiled code-based extensions). Most of the changes needed for scons have been integrated upstream, though, except one or two things. I see. I think it's a bit confusing that one needs to build a new build system just to build numpy, e.g. that both distutils and scons are not good enough. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] porting NumPy to Python 3
Hi James, On Thu, Jan 29, 2009 at 2:11 AM, James Watson watson@gmail.com wrote: Hi, I am interested in contributing to the port of NumPy to Python 3. Who I should coordinate effort with? I have started at the Python end of the problem (as opposed to http://www.scipy.org/Python3k), e.g. I have several patches to get 2to3 to work on NumPy's Python source code. I am sorry that noone has replied to your email. Could you please upload your patches somewhere? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] whohas command
Hi, I just discovered a new whohas command in Debian, that can show a package in virtually all important distributions (Arch, Debian, Fedora, Gentoo, openSUSE, Slackware (and linuxpackages.net), Source Mage, Ubuntu, FreeBSD, NetBSD, OpenBSD, Fink and MacPorts distributions). So for example SymPy is in a lot of them already: $ whohas sympy Archpython-sympy-git 20090121-1 unsupported http://aur.archlinux.org/packages.php?ID=23341 MacPortspy-sympy 0.5.15 http://trac.macports.org/browser/trunk/dports/python/py-sympy/Portfile MacPortspy25-sympy0.6.3 http://trac.macports.org/browser/trunk/dports/python/py25-sympy/Portfile FreeBSD py25-sympy0.6.3 math http://www.freebsd.org/cgi/pds.cgi?ports/math/py25-sympy NetBSD py-sympy 0.5.15 mathhttp://pkgsrc.se/math/py-sympy Ubuntu python-sympy 0.4.2-1 universehttp://packages.ubuntu.com/gutsy/python-sympy Ubuntu python-sympy 0.5.8-1 universehttp://packages.ubuntu.com/hardy/python-sympy Ubuntu python-sympy 0.5.15-1~hardy1 universe http://packages.ubuntu.com/hardy-backports/python-sympy Ubuntu python-sympy 0.5.15-1 universehttp://packages.ubuntu.com/intrepid/python-sympy Ubuntu python-sympy 0.6.3-1 universehttp://packages.ubuntu.com/jaunty/python-sympy Debian python-sympy 0.6.1-1 testing http://packages.debian.org/lenny/python-sympy Finksympy-py240.5.6-1 http://pdb.finkproject.org/pdb/package.php/sympy-py24 Finksympy-py250.5.6-1 http://pdb.finkproject.org/pdb/package.php/sympy-py25 Gentoo sympy 0.6.3 http://gentoo-portage.com/dev-python/sympy Gentoo sympy 0.6.2 http://gentoo-portage.com/dev-python/sympy openSUSEpython-sympy 0.6.3 science http://packages.opensuse-community.org/packageinfo.jsp?checksum=a2258d3deaee5f411456f22ea1ab6bb7ec95e12camp;distro=openSUSE_111 It's also interesting to click on the links to get to the pages of sympy for each distribution. And numpy: $ whohas numpy Archpython-numexpr1.1.1-1 unsupported http://aur.archlinux.org/packages.php?ID=23380 Archpython-numpy 1.2.1-3 community http://aur.archlinux.org/packages.php?ID=8497 Archscikits-audiolab-svn 1491-1 unsupported http://aur.archlinux.org/packages.php?ID=21018 FreeBSD py25-numpy1.2.1,1 math http://www.freebsd.org/cgi/pds.cgi?ports/math/py25-numpy FreeBSD py25-symeig 1.4_1 math http://www.freebsd.org/cgi/pds.cgi?ports/math/py25-symeig NetBSD py-numpy 1.1.0 wip http://pkgsrc.se/wip/py-numpy NetBSD py-numpy 1.1.0 mathhttp://pkgsrc.se/math/py-numpy Ubuntu python-numpy 1:1.0.1-8ubuntu1 universehttp://packages.ubuntu.com/feisty/python-numpy Ubuntu python-numpy 1:1.0.3-1ubuntu2 universehttp://packages.ubuntu.com/gutsy/python-numpy Ubuntu python-numpy 1:1.0.4-6ubuntu3 universehttp://packages.ubuntu.com/hardy/python-numpy Ubuntu python-numpy 1:1.1.1-1 http://packages.ubuntu.com/intrepid/python-numpy Ubuntu python-numpy 1:1.1.1-2ubuntu1 http://packages.ubuntu.com/jaunty/python-numpy Ubuntu python-numpy-dbg 1:1.0.1-8ubuntu1 universehttp://packages.ubuntu.com/feisty/python-numpy-dbg Ubuntu python-numpy-dbg 1:1.0.3-1ubuntu2 universehttp://packages.ubuntu.com/gutsy/python-numpy-dbg Ubuntu python-numpy-dbg 1:1.0.4-6ubuntu3 universehttp://packages.ubuntu.com/hardy/python-numpy-dbg Ubuntu python-numpy-dbg 1:1.1.1-1 http://packages.ubuntu.com/intrepid/python-numpy-dbg Ubuntu python-numpy-dbg 1:1.1.1-2ubuntu1 http://packages.ubuntu.com/jaunty/python-numpy-dbg Ubuntu python-numpy-dev 1:1.0.1-8ubuntu1 universehttp://packages.ubuntu.com/feisty/python-numpy-dev Ubuntu python-numpy-dev 1:1.0.3-1ubuntu2 universehttp://packages.ubuntu.com/gutsy/python-numpy-dev Ubuntu python-numpy-dev http://packages.ubuntu.com/hardy/python-numpy-dev Ubuntu python-numpy-doc 1:1.0.1-8ubuntu1 universehttp://packages.ubuntu.com/feisty/python-numpy-doc Ubuntu python-numpy-doc 1:1.0.3-1ubuntu2 universehttp://packages.ubuntu.com/gutsy/python-numpy-doc Ubuntu python-numpy-doc 1:1.0.4-6ubuntu3 universehttp://packages.ubuntu.com/hardy/python-numpy-doc Ubuntu python-numpy-doc 1:1.1.1-1
Re: [Numpy-discussion] missing doc dir in the official tarball
On Sat, Dec 20, 2008 at 3:02 PM, Pauli Virtanen p...@iki.fi wrote: Sat, 20 Dec 2008 20:15:43 +0900, David Cournapeau wrote: On Sat, Dec 20, 2008 at 7:43 PM, Ondrej Certik ond...@certik.cz wrote: Just to make it clear -- I think the docs should not be generated in the tarball -- only the sources should be there. I agree this makes more sense for you, as a packager, but I am not sure it makes much sense to put the doc sources in the tarball for users (Building numpy should only require python + a C compiler; building the doc is more difficult -you need at least sphinx and all its dependencies). For audiolab, I put the generated doc, thinking if people want to mess with the doc, they are knowledgeable enough to deal with svn - but I did not think about the packagers :) I am not sure what's the best solution: maybe put both in the (released) source tarball ? I'd say that we put the source for the documentation to the documentation tarball, and distribute the built HTML+whatever documentation in a separate package. Why not to just include the *sources* together with numpy, and possibly include html+whatever in a separate documentation package? That way everybody is happy. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] missing doc dir in the official tarball
On Sun, Dec 21, 2008 at 1:49 PM, Pauli Virtanen p...@iki.fi wrote: Sun, 21 Dec 2008 13:05:57 +0100, Ondrej Certik wrote: On Sat, Dec 20, 2008 at 3:02 PM, Pauli Virtanen p...@iki.fi wrote: Sat, 20 Dec 2008 20:15:43 +0900, David Cournapeau wrote: On Sat, Dec 20, 2008 at 7:43 PM, Ondrej Certik ond...@certik.cz wrote: Just to make it clear -- I think the docs should not be generated in the tarball -- only the sources should be there. I agree this makes more sense for you, as a packager, but I am not sure it makes much sense to put the doc sources in the tarball for users (Building numpy should only require python + a C compiler; building the doc is more difficult -you need at least sphinx and all its dependencies). For audiolab, I put the generated doc, thinking if people want to mess with the doc, they are knowledgeable enough to deal with svn - but I did not think about the packagers :) I am not sure what's the best solution: maybe put both in the (released) source tarball ? I'd say that we put the source for the documentation to the documentation tarball, and distribute the built HTML+whatever documentation in a separate package. Why not to just include the *sources* together with numpy, and possibly include html+whatever in a separate documentation package? That's what I tried to say, but mistyped source as documentation. Ok, so we all seem to agree that having (at least) the source of docs together with the main numpy tarball is a good thing. I'll try to have a look at this. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] missing doc dir in the official tarball
On Sat, Dec 20, 2008 at 3:35 AM, David Cournapeau courn...@gmail.com wrote: On Sat, Dec 20, 2008 at 7:43 AM, Stéfan van der Walt ste...@sun.ac.za wrote: 2008/12/20 Ondrej Certik ond...@certik.cz: So we thought with Stefan that maybe a simpler solution is just to fix the ./setup sdist (or how you create the tarball in numpy) to include documentation and be done with it. I think releases should either include the Sphinx documentation or, alternatively, we should provide a separate tar-ball for the docs along with every release. How difficult would it be to generate the doc ? I have not followed in detail what happened recently on that front. Is sphinx + sphinx ext the only necessary additional tools ? Just to make it clear -- I think the docs should not be generated in the tarball -- only the sources should be there. The html (and/or pdf) docs will be generated at the package build. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Missing numpy.i
On Tue, Nov 18, 2008 at 11:42 AM, Nicolas ROUX nicolas.r...@st.com wrote: Hi, About the missing doc directory in the windows install in latest numpy release, could you please add it ? (please see below the previous thread) Well, this is a serious problem, so it should definitely be fixed, see here: http://projects.scipy.org/pipermail/numpy-discussion/2008-December/039309.html Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] python-numpy: memory leak in exponentiation
Hi, the details including a test script are at: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=505999 Thanks, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Fri, Sep 19, 2008 at 10:37 AM, Robert Cimrman [EMAIL PROTECTED] wrote: Ondrej Certik wrote: On Thu, Sep 18, 2008 at 4:12 PM, Ryan May [EMAIL PROTECTED] wrote: Ondrej Certik wrote: On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman [EMAIL PROTECTED] wrote: It requires Cython and a C compiler to build. I'm still debating myself about the desired workflow for using it, but for now, it only profiles functions which you have registered with it. I have made the profiler work as a decorator to make this easy. E.g., many thanks for this! I have wanted to try out the profiler but failed to build it (changeset 6 0de294aa75bf): $ python setup.py install --root=/home/share/software/ running install running build running build_py creating build creating build/lib.linux-i686-2.4 copying line_profiler.py - build/lib.linux-i686-2.4 running build_ext cythoning _line_profiler.pyx to _line_profiler.c building '_line_profiler' extension creating build/temp.linux-i686-2.4 i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c -o build/temp.linux-i686-2.4/_line_profiler.o _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a function) error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. I am telling you all the time Robert to use Debian that it just works and you say, no no, gentoo is the best. :) And what's wrong with that? :) Once you get over the learning curve, Gentoo works just fine. Must be Robert K.'s fault. :) Well, I think if Robert C. hasn't yet get over the learning curve after so many years of hard work, maybe the learning curve is too steep. :) This is most probably not related to Gentoo at all and certainly not related to me knowing Gentoo or not :) (and no, learning Gentoo is not that hard.) Let us know where the problem was. :) I am just using common sense, if something works on Debian and macosx and doesn't work on gentoo, I thought it was safe to say it was gentoo related, but I may well be wrong. :)) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Thu, Sep 18, 2008 at 1:29 AM, Robert Kern [EMAIL PROTECTED] wrote:
On Wed, Sep 17, 2008 at 18:09, Ondrej Certik [EMAIL PROTECTED] wrote:
On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern [EMAIL PROTECTED] wrote:
On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg [EMAIL PROTECTED] wrote:
That would make me an extremely happy user, I've been looking for this for
years!
I can't imagine I'm the only one who profiles some hundred lines of code
and
ends up with 90% of total time in the dot-function
For the time being, you can grab it here:
http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/
It requires Cython and a C compiler to build. I'm still debating
myself about the desired workflow for using it, but for now, it only
profiles functions which you have registered with it. I have made the
profiler work as a decorator to make this easy. E.g.,
from line_profiler import LineProfiler
profile = LineProfiler()
@profile
def my_slow_func():
...
profile.dump_stats('my_slow_func.lprof')
This is kind of inconvenient, so I have a little script that handles
everything except for the @profile itself. It started as a script to
run cProfile nicely, so it actually does that by default.
I took pystone.py from the Python source and added a couple of
@profile decorators to demonstrate:
[line_profiler]$ ./kernprof.py --help
Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg]
...
Options:
-h, --helpshow this help message and exit
-l, --line-by-lineUse the line-by-line profiler from the line_profiler
module instead of cProfile. Implies --builtin.
-b, --builtin Put 'profile' in the builtins. Use 'profile.enable()'
and 'profile.disable()' in your code to turn it on
and
off, or '@profile' to decorate a single function, or
'with profile:' to profile a single section of code.
-o OUTFILE, --outfile=OUTFILE
Save stats to outfile
-s SETUP, --setup=SETUP
Code to execute before the code to profile
[line_profiler]$ ./kernprof.py -l pystone.py
Pystone(1.1) time for 5 passes = 11.34
This machine benchmarks at 4409.17 pystones/second
Wrote profile results to pystone.py.lprof
[line_profiler]$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 8.46516 s
[...]
This is what I am getting:
$ ./kernprof.py -l pystone.py
Wrote profile results to pystone.py.lprof
$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
$
So I think you meant:
$ ./kernprof.py -l mystone.py
20628
Wrote profile results to mystone.py.lprof
$ ./view_line_prof.py mystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 13.0803 s
[...]
Now it works.
No, I meant pystone.py. My script-finding code may have (incorrectly)
found a different, uninstrumented pystone.py file somewhere else,
though. Try with ./pystone.py.
This is an excellent piece of software! Nice job. Just today I needed
such a thing!
How do you easily install it?
python setup.py install should have installed the module. I haven't
done anything with the scripts, yet.
I usually do python setup.py install
--home=~/lib and I have the PYTHONPATH + PATH setup in my .bashrc,
but I then need to manually remove the stuff from my ~/lib if I want
to uninstall, which sucks. So this time I just did python setup.py
build and moved the .so file manually to the current dir. But there
must be a better way. What is your workflow?
For things I am developing on, I build them in-place, using the
setuptools develop command to add the appropriate path to the
easy-install.pth file. To remove, I would just edit that file.
For thing I'm not developing on, I usually build and install an egg if
at all possible.
But then, I'm typically on a single-user box where I'm root, so I
sometimes do nasty and unsanitary things like chown -R rkern:rkern
/usr/local/.
Anyway, so I used it on my code and here is what I got:
File: hermes1d/mesh.py
Function: integrate_function at line 119
Total time: 0.647412 s
Line # Hits Time % Time Line Contents
=
119 @profile
120 def integrate_function(self, f):
121
122 Integrate the function
f on the element.
123
12496 1091 0.2 from numpy import array
12596 461070 71.2 from scipy.integrate
import quadrature
12696 496 0.1 a, b =
self.nodes[0].x, self.nodes[1].x
12796 418 0.1
Re: [Numpy-discussion] profiling line by line
On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman [EMAIL PROTECTED] wrote: Hi Robert, Robert Kern wrote: On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg [EMAIL PROTECTED] wrote: That would make me an extremely happy user, I've been looking for this for years! I can't imagine I'm the only one who profiles some hundred lines of code and ends up with 90% of total time in the dot-function For the time being, you can grab it here: http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ It requires Cython and a C compiler to build. I'm still debating myself about the desired workflow for using it, but for now, it only profiles functions which you have registered with it. I have made the profiler work as a decorator to make this easy. E.g., many thanks for this! I have wanted to try out the profiler but failed to build it (changeset 6 0de294aa75bf): $ python setup.py install --root=/home/share/software/ running install running build running build_py creating build creating build/lib.linux-i686-2.4 copying line_profiler.py - build/lib.linux-i686-2.4 running build_ext cythoning _line_profiler.pyx to _line_profiler.c building '_line_profiler' extension creating build/temp.linux-i686-2.4 i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c -o build/temp.linux-i686-2.4/_line_profiler.o _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a function) error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. I am telling you all the time Robert to use Debian that it just works and you say, no no, gentoo is the best. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
So the timing raises a lot. For obvious reasons, that's the overhead of the profiler. But the problem is that then the timings just don't fit, e.g. if I sum the total time spent in subfunctions, it doesn't account for all the time printed on the respective line in the parent function. I don't know if there is any way to fix it, or even worth fixing. So I guess one should just use the profiling info to determine which line to fix. Do you think it's worthy to implement line profiling for all lines and then make sure that all timings match? What is your experience. The reason I want this is so that I can determine just by looking at the timing how much time is spent at the line and how much time is spent at the functions that are being called at the line. I think it is doable, what do you think? One would trace how much time was wasted in python_trace_callback and then just substract this time from all parent function's lines timings. Btw Robin, how is Matlab doing it? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern [EMAIL PROTECTED] wrote:
On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg [EMAIL PROTECTED] wrote:
That would make me an extremely happy user, I've been looking for this for
years!
I can't imagine I'm the only one who profiles some hundred lines of code and
ends up with 90% of total time in the dot-function
For the time being, you can grab it here:
http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/
It requires Cython and a C compiler to build. I'm still debating
myself about the desired workflow for using it, but for now, it only
profiles functions which you have registered with it. I have made the
profiler work as a decorator to make this easy. E.g.,
from line_profiler import LineProfiler
profile = LineProfiler()
@profile
def my_slow_func():
...
profile.dump_stats('my_slow_func.lprof')
This is kind of inconvenient, so I have a little script that handles
everything except for the @profile itself. It started as a script to
run cProfile nicely, so it actually does that by default.
I took pystone.py from the Python source and added a couple of
@profile decorators to demonstrate:
[line_profiler]$ ./kernprof.py --help
Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ...
Options:
-h, --helpshow this help message and exit
-l, --line-by-lineUse the line-by-line profiler from the line_profiler
module instead of cProfile. Implies --builtin.
-b, --builtin Put 'profile' in the builtins. Use 'profile.enable()'
and 'profile.disable()' in your code to turn it on and
off, or '@profile' to decorate a single function, or
'with profile:' to profile a single section of code.
-o OUTFILE, --outfile=OUTFILE
Save stats to outfile
-s SETUP, --setup=SETUP
Code to execute before the code to profile
[line_profiler]$ ./kernprof.py -l pystone.py
Pystone(1.1) time for 5 passes = 11.34
This machine benchmarks at 4409.17 pystones/second
Wrote profile results to pystone.py.lprof
[line_profiler]$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 8.46516 s
[...]
This is what I am getting:
$ ./kernprof.py -l pystone.py
Wrote profile results to pystone.py.lprof
$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
$
So I think you meant:
$ ./kernprof.py -l mystone.py
20628
Wrote profile results to mystone.py.lprof
$ ./view_line_prof.py mystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 13.0803 s
[...]
Now it works.
This is an excellent piece of software! Nice job. Just today I needed
such a thing!
How do you easily install it? I usually do python setup.py install
--home=~/lib and I have the PYTHONPATH + PATH setup in my .bashrc,
but I then need to manually remove the stuff from my ~/lib if I want
to uninstall, which sucks. So this time I just did python setup.py
build and moved the .so file manually to the current dir. But there
must be a better way. What is your workflow?
Anyway, so I used it on my code and here is what I got:
File: hermes1d/mesh.py
Function: integrate_function at line 119
Total time: 0.647412 s
Line # Hits Time % Time Line Contents
=
119 @profile
120 def integrate_function(self, f):
121
122 Integrate the function
f on the element.
123
12496 1091 0.2 from numpy import array
12596 461070 71.2 from scipy.integrate
import quadrature
12696 496 0.1 a, b =
self.nodes[0].x, self.nodes[1].x
12796 418 0.1 def func(x):
128 #print x
129 #print
array([f.f(y) for y in x])
130 return
array([f.f(y) for y in x])
13196 183903 28.4 val, err =
quadrature(func, a, b)
132 #print val, a, b
133 #stop
13496 434 0.1 return val
This is interesting, because 70% of time is spent on from
scipy.integrate import quadrature. So this is not good.
So I moved it out and voila! My code is 3.4x faster. I really didn't
know that importing (scipy.integrate) is *so* slow.
File: hermes1d/mesh.py
Function: integrate_function at line 121
Total time: 0.189429 s
Line # Hits Time % Time Line Contents
Re: [Numpy-discussion] Updated Numpy reference guide
Hi Pauli! On Sun, Aug 31, 2008 at 8:21 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: Hi all, I finished the first iteration of incorporating material from Travis Oliphant's Guide to Numpy to the Sphinxy reference guide we were constructing in the Doc marathon. Result is here: (the PDF is a bit ugly, though, some content is almost randomly scattered there) http://www.iki.fi/pav/tmp/numpy-refguide/index.xhtml http://www.iki.fi/pav/tmp/numpy-refguide/NumPy.pdf Source is here: (Stéfan, if it looks ok to you, could you pull and check if it builds for you when you have time?) https://code.launchpad.net/~pauli-virtanen/scipy/numpy-refguide What I did with the Guide to Numpy material was: - Collapsed each of the reference Chapters 3, 6, 8, 9 (ndarrays, scalars, dtypes, ufuncs) with the more introductory material in Chapter 2. - As this was supposed to be a reference guide, I tried to compress the text from Chapter 2 as much as possible, by sticking to definitions and dropping some more tutorial-oriented parts. This may have reduced readability at some points... - I added some small bits or rewrote parts in the above sections in places where I thought it would improve the result. - I did not include material that I thought was better to be put into appropriate docstrings in Numpy. What to do with class docstrings and obscure __xxx__ attributes was not so clear a decision, so what I did for these varies. - The sections about Ufuncs and array indexing are taken almost verbatim from the Guide to Numpy. The ndarray, scalar and dtype sections somewhat follow the structure of the Guide, but the text is more heavily edited from the original. Some things to do: - Descriptions about constructing items with __new__ methods should probably still be clarified; I just replaced references to __new__ with references to the corresponding classes. - What to do with the material from numpy.doc.* should be decided, as the text there doesn't look like it should go into a reference manual. Some questions: - Is this good enough to go into Numpy SVN at some point? Or should we redo it and base the work closer to the original Guide to Numpy? - Does it build for you? (I'd recommend using the development 0.5 version of Sphinx, so that you get the nifty Inter-Sphinx links to the Python documentation.) We are unfortunately beating the Sphinx with a big stick to make it place the documentation of each function or class into a separate file, and to convert the Numpy docstring format to something the Sphinx can fathom. :) There's also some magic in place to make toctrees:: of function listings more pleasant to the eye. Any comments of what should be improved are welcome. (Even better: clone the bzr branch, make the changes yourself, and put the result somewhere available! E.g. as a bzr bundle or a branch on the launchpad.) I think it looks excellent. It'd be cool if all the docs could finally be at one place, instead of scattered all over the wiki. So for me, any form in sphinx is ok. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy08] Documentation BoF
On Fri, Aug 22, 2008 at 10:26 AM, Stéfan van der Walt [EMAIL PROTECTED] wrote: Hi all, This is my personal recollection of the documentation BoF. Feel free to comment or correct the text below. Regards Stéfan Summary of the Documentation Birds-of-a-Feather Session === Topics proposed for discussion -- - Path towards getting SciPy documented using the online documentation editor - Intersphinx: linking Sphinx-using projects - Culture / methodology: what do we expect from code accepted into NumPy/SciPy? - Tools for reading documentation (Help 2.0) - The complete SciPy book -- how to get there - NumPy Documentation Standard: is it useful beyond NumPy? Conclusions reached --- Documentation format - Question: could we standardise the descriptions of data-types between docs of different projects? They should overlap to some extent. - If we could agree on some common standard (or at least a common subset of a format) across projects, IPython could parse and display docstrings with special markup. - Matplotlib should be involved in this conversation. They have already written a vast amount of documentation, and their needs may be different to those of the SciPy projects. Formal requirements ``` For functions going into NumPy and SciPy, we have a certain minimum documentation requirements. However, we'd rather take a somewhat liberal, open approach, and accept code with somwhat inferior docstrings, working with the author to improve them. This way, we remain accessible as a community, while striving to improve the quality of the code base. Marketing SciPy ``` The current entry point for SciPy on the web (http://www.scipy.org) may be confusing to beginners. The web page should be redesigned, in a similar fashion as the new Sage and code.enthought.com pages, linking to the messy wiki behind it in a consistent fashion. Joe Harrington may be able to hire someone to work on this important aspect of SciPy marketing. Cross-project documentation searching ` Ideally, one should be able to perform a documentation search across several projects. In order to achieve this, we should negotiate a common convention for installing Sphinx-generated documentation into a standard location. A javascript-backed webpage can then be provided to do a search, using the json indices, or some other means. +1 Currently sphinx can't handle scipy docstrings, can it? It didn't for me at least. It'd be nice if whatever format you agre upon, could work with sphinx's autodoc. Also I am very interested in your web based editing of docstrings, so that people can just fix and improve the docstrings easily. I can imagine something like In [1]: e.series??? in ipython and it would fireup a browser pointing to the docstring and one would just fix it. Since you did most of the work already, maybe the above won't be that hard to do either. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible new multiplication operators for Python
On Tue, Aug 19, 2008 at 1:57 PM, Alan G Isaac [EMAIL PROTECTED] wrote: On Sun, Aug 17, 2008 at 04:28:55PM -0400, Alan G Isaac wrote: That said, what kind of problems do you have in mind? Gael Varoquaux wrote: wht I am most worried about is not being able to enter the symbol, because I am in an editor I don't know, and the symbol is not on my keyboard. This has happened to me more than once with unicode. Well anyone who travels ends up dealing with strange keyboards, I suppose, but I would hate for the possibility that someone is in the odd situation of this happening while not having access to the net (and thus to Vim or emacs) dictate the decision on this. That would just not be very forward looking, especially when one can *in this case* always use ``dot`` instead. I absolutely agree with Gael here. Using anything besides lower (128) ascii is imho not wise. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible new multiplication operators for Python
On Mon, Aug 18, 2008 at 1:50 AM, Andrew Dalke [EMAIL PROTECTED] wrote: On Aug 18, 2008, at 12:00 AM, Ondrej Certik wrote: There is some inconsistency though, for example one can override A() + A(), but one cannot override 1 + 1. This could (should) be fixed somehow. That will never, ever change in Python. There's no benefit to being able to redefine int.__add__ and doing so will break entirely too many assumptions. Here's one assumption - the C implementation does some simple constant folding: def f(): ... print 1+1 ... import dis dis.dis(f) 2 0 LOAD_CONST 2 (2) 3 PRINT_ITEM 4 PRINT_NEWLINE 5 LOAD_CONST 0 (None) 8 RETURN_VALUE With what you want that's not possible. Just think of the implementation difficulty. Are changes on the per-module or per-scope or per-thread level? And performance would be lousy (or require deep inferencing analysis) because every operation in C would need to go through the Python API just in case some fundamental definition like this was changed. Such a language is possible. I wouldn't be surprised if you could do it in Smalltalk and mostly do it in Ruby. But the general belief is that good design follows the open/closed principle: software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification - Bertrand Meyer, quoted by http://en.wikipedia.org/wiki/Open/closed_principle In Python, all types are closed for modification, and while classes are open for modification it's highly frowned upon to do so. The name monkey-patch sounds somewhat derisive for a reason. Yeah, I understand the reasoning. My reason is that sometimes you want to do something else on 1/2 rather than to get 0, or 0.500. I would like to get my own class called, but if it's again Python, then I am perfectly happy with Python as it is now. No changes needed. Anyway, this is off topic. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
Hi, with Andrew's permission, I am starting a new thread, where our discussion is ontopic. :) My original question was, that I would like to override 1+1 to return MyClass(1, 1) or something. Robert said it would break other libraries and Andrew said this: On Mon, Aug 18, 2008 at 9:23 PM, Andrew Dalke [EMAIL PROTECTED] wrote: There are basically two possible options: 1) global overloading of + 2) allow to tell Python that 1 is not its int, but ours Integer. BTW, Sage just preparses Python for exactly this reason and substitutes all numbers like 1 with Integer(1). This really sucks imho. How would you like to do it? Any solution I can think of would cause huge problems. For example, being able to change int.__add__ would cause systemic problems throughout the code base and would prevent, or very much hinder, migration of Python code to C. Agree. Changes to support a different constructor for basic types, on a per-module basis has its own problems. When are the new types specified? In the module itself or by the code which imports the module? Can the definitions be changed on a scope-by-scope level? Now, I can imagine a Python with builtin multimethod dispatch defined via static scoping, so that def __add__(left:int, right:int): return __builtin__.__add__(__builtin__.__add__(left, right), 2) def __int__(s): return Decimal(s) might work, but even here there's a problem because the 2 in __add__ gets replaced by a Decimal, when I really want it to be an integer. Not getting why is 2 replaced by Decimal if you don't want to? If you don't want it, you can just reimplement __int__(s). This would of course be only active in the module where it was defined. So I don't see how you can do it in the context of Python. In the context of a very different programming language, sure, it's possible. But that's what Sage is doing, they just preparse the input. But in SymPy we'd like to use it as a regular Python library in users programs, so imho preparsing, or custom compiling using your package is not really an option, is it? Use a different extension, like .sympy and an import hook which does the .sympy - .pyc conversion. But it's not one I recommend for real code, at least not without a lot of personal experience to see if it's worthwhile in the face of the downsides. Yes, this is a mess, this is just like preparsing. Well, not like -- this is preparsing. Ok, in the current state, you don't know either what's going to happen. If you write In [1]: x/2*3/4 you have no idea what the result is going to be, you need to analyze x.__div__() and start from there. But if you write In [2]: 1/2*3/4 currently you know it will be 0. But imho you could as well analyze the global __mul__ (or global __int__, depending on how this would be technically implemented) to see what's going to happen. I mean what is the difference between [1] and [2]? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
On Mon, Aug 18, 2008 at 10:45 PM, Andrew Dalke [EMAIL PROTECTED] wrote: On Aug 18, 2008, at 10:01 PM, Ondrej Certik wrote: with Andrew permission, I am starting a new thread, where our discussion is ontopic. :) Though I want to point out that without specific proposals of how the implementation might look, this thread will not go anywhere as it will be too distant from usable code. I sent examples to show how such a system might look, as the basis for getting a feel if it was practical. I do not think my examples are practical, but they were meant as an example of how such a proposal might look. Since I know that the Python implementation will not change to support per-module or per-scope redefinitions for 1+2 and builtin object constructors, the only feasible mechanism is through some sort of alternate grammar that compiles to either Python or directly to the Python virtual machine. One such was is through import hooks. Yes, this is a mess, this is just like preparsing. Well, not like -- this is preparsing. It's not preparsing. It's parsing. There's no pre about it. It's not a macro language. My ply4python tutorial compiles various Python-like languages to the Python virtual machine bytecode. I mean what is the difference between [1] and [2]? I want to see how you would extend Python to support such a mechanism before I worried about how to interpret it. Or in other words, the difference between [1] and [2] is that [2] can be fully evaluated through simple static analysis, while [1] cannot. BTW, this is unexpected. Python does constant folding of that expression, but only with specific settings. import dis def f(): ... print 1/2*3/4 ... dis.dis(f) 2 0 LOAD_CONST 1 (1) 3 LOAD_CONST 2 (2) 6 BINARY_DIVIDE 7 LOAD_CONST 3 (3) 10 BINARY_MULTIPLY 11 LOAD_CONST 4 (4) 14 BINARY_DIVIDE 15 PRINT_ITEM 16 PRINT_NEWLINE 17 LOAD_CONST 0 (None) 20 RETURN_VALUE from __future__ import division def f(): ... print 1/2*3/4 ... dis.dis(f) 2 0 LOAD_CONST 7 (0.375) 3 PRINT_ITEM 4 PRINT_NEWLINE 5 LOAD_CONST 0 (None) 8 RETURN_VALUE The only way I can see to do what you want requires multimethods, which don't currently exist in Python except as third-party extensions. The one I know about, from Philip J. Eby, works on a global-level, not module level, because of how registration happens, so it does not support what you would like. One way to fix that would be to use the trick that Travis Oliphant told me at EuroSciPy -- hack the while (or if) statement and do the preparsing in there. So clearly the language seems to support that, so imho it could be made more official. E.g. you would write: while sympy: e = 1/2 and e would be Integer(1)/Integer(2). But anyway, it's kind of hackish and I don't know what to say more about it, besides what I already said. The problem is that I don't have time to dig more into Python internals and without it it seems I cannot provide more constructive answer besides I want some way to avoid Python reducing 1/2 to 0. Generally I believe that if there is will, it can always be done. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
Hi Christian, On Mon, Aug 18, 2008 at 11:22 PM, Christian Heimes [EMAIL PROTECTED] wrote: Ondrej Certik wrote: Ok, in the current state, you don't know either what's going to happen. If you write In [1]: x/2*3/4 you have no idea what the result is going to be, you need to analyze x.__div__() and start from there. But if you write In [2]: 1/2*3/4 currently you know it will be 0. But imho you could as well analyze the global __mul__ (or global __int__, depending on how this would be technically implemented) to see what's going to happen. I mean what is the difference between [1] and [2]? Andrew has already pointed it out very well. I like to comment on your proposal from the perspective of a Python core developer as well as the perspective of somebody who has worked with Guido for more than a year. I'd bet my life that Guido is never every going to allow it. The core types are fundemental to the Python interpreter. Even the possibility of pluggable type methods would make the implementation slower, more fragile and much more complicated. We'd have to remove several speed tricks and special cases for e.g. ints and replace them with slower implementations. But don't give up hope yet! During the alpha phase of Python 3.0 and the revamping of the decimal module, some core developers had an even better idea. We were discussing the possibility of making decimals the default for float literals. The idea was rejected eventually but it gave birth to yet another idea. What about making the *result* of a literal pluggable? The Python creates a float for the literal 1.0. Some header in a module could replace the default target 'float' with e.g. decimal.Decimal. Example syntax (rough idea): type(1.0) type 'float' with float as from decimal import Decimal type(1.0) class 'decimal.Decimal' Wouldn't that solve your general problem more elegant without breaking other modules? It absolutely would. Thanks very much for the email. How is your proposal (redefine literals) different to just saying to Python -- hey, just call my class when someone writes 1, e.g. proposition 2) from my first email? Or am I missing something. I agree with the technical reasonings, why some particular solution is not good. I.e. I didn't do any proposal, I am just trying to find a way, so that we don't have to always type In [3]: Integer(1)/2 * x sometimes, but In [4]: x/2 some other times. If you know what I mean. Both do the same thing, but [1] is very annoying to write and a source of common mistakes that people do with SymPy, it simply returns 0. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
On Tue, Aug 19, 2008 at 1:06 AM, Christian Heimes [EMAIL PROTECTED] wrote: Andrew Dalke wrote: When would this with float ... considered valid? [long posting] Oh h... what have I done ... *g* Slow down, please. For now there are no concrete plans what-so-ever to implement the feature in the near future. Some developers have expressed their interest in a way to alter the resulting type of a literal. It was my attention to show you, that we have discussed the idea, too. Now for the with type as from import syntax. I came up with the syntax idea about an hour ago. I tried to come up with some nice syntax that reuses existing keywords. IMHO it has a nice ring. Other possibilities I came up with: def float as factory def float as from module import factory with float yield factory with float yield from module import factory After some careful thinking I'm in favor of with ... yield It's less ambiguous and can't be mistaken for with open(filename) as fh. The ideas needs a good PEP. You are definitely up to something. You also came up with a list of possible issues and corner cases. Are you interested in pursuing the proposal? *wink* Are we able to provide an actual patch to Python that implements this? If so, then I am. Imho the proposal should come with an actual patch, otherwise it's difficult to judge it. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible new multiplication operators for Python
On Sun, Aug 17, 2008 at 7:03 AM, Fernando Perez [EMAIL PROTECTED] wrote: Hi all, [ please keep all replies to this only on the numpy list. I'm cc'ing the scipy ones to make others aware of the topic, but do NOT reply on those lists so we can have an organized thread for future reference] In the Python-dev mailing lists, there were recently two threads regarding the possibility of adding to the language new multiplication operators (amongst others). This would allow one to define things like an element-wise and a matrix product for numpy arrays, for example: http://mail.python.org/pipermail/python-dev/2008-July/081508.html http://mail.python.org/pipermail/python-dev/2008-July/081551.html It turns out that there's an old pep on this issue: http://www.python.org/dev/peps/pep-0225/ which hasn't been ruled out, simply postponed. At this point it seems that there is room for some discussion, and obviously the input of the numpy/scipy crowd would be very welcome. I volunteered to host a BOF next week at scipy so we could collect feedback from those present, but it's important that those NOT present at the conference can equally voice their ideas/opinions. So I wanted to open this thread here to collect feedback. We'll then try to have the bof next week at the conference, and I'll summarize everything for python-dev. Obviously this doesn't mean that we'll get any changes in, but at least there's interest in discussing a topic that has been dear to everyone here. I like that Python is so easy to learn, so I hope no more operators and definitely not unicode operators will be added. And if, then only if they are really needed, which I think they are not. What I like on Python is that I can remember all it's syntax in my memory. The more operators, the more difficult this will be. There is some inconsistency though, for example one can override A() + A(), but one cannot override 1 + 1. This could (should) be fixed somehow. But apart from that, I very much like Python as it is now and I hope it will not become a bloated language. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
On Thu, Jul 31, 2008 at 10:02 PM, Robert Kern [EMAIL PROTECTED] wrote:
On Thu, Jul 31, 2008 at 05:43, Andrew Dalke [EMAIL PROTECTED] wrote:
On Jul 31, 2008, at 12:03 PM, Robert Kern wrote:
But you still can't remove them since they are being used inside
numerictypes. That's why I labeled them internal utility functions
instead of leaving them with minimal docstrings such that you would
have to guess.
My proposal is to replace that code with a table mapping
the type name to the uppercase/lowercase/capitalized forms,
thus eliminating the (small) amount of time needed to
import string.
It makes adding new types slightly more difficult.
I know it's a tradeoff.
Probably not a bad one. Write up the patch, and then we'll see how
much it affects the import time.
I would much rather that we discuss concrete changes like this rather
than rehash the justifications of old decisions. Regardless of the
merits about the old decisions (and I agreed with your position at the
time), it's a pointless and irrelevant conversation. The decisions
were made, and now we have a user base to whom we have promised not to
break their code so egregiously again. The relevant conversation is
what changes we can make now.
Some general guidelines:
1) Everything exposed by from numpy import * still needs to work.
a) The layout of everything under numpy.core is an implementation detail.
b) _underscored functions and explicitly labeled internal functions
can probably be modified.
c) Ask about specific functions when in doubt.
2) The improvement in import times should be substantial. Feel free to
bundle up the optimizations for consideration.
3) Moving imports from module-level down into the functions where they
are used is generally okay if we get a reasonable win from it. The
local imports should be commented, explaining that they are made local
in order to improve the import times.
4) __import__ hacks are off the table.
5) Proxy objects ... I would really like to avoid proxy objects. They
have caused fragility in the past.
6) I'm not a fan of having environment variables control the way numpy
gets imported, but I'm willing to consider it. For example, I might go
for having proxy objects for linalg et al. *only* if a particular
environment variable were set. But there had better be a very large
improvement in import times.
I just want to say that I agree with Andrew that slow imports just
suck. But it's not really that bad, for example on my system:
In [1]: %time import numpy
CPU times: user 0.11 s, sys: 0.01 s, total: 0.12 s
Wall time: 0.12 s
so that's ok. For comparison:
In [1]: %time import sympy
CPU times: user 0.12 s, sys: 0.02 s, total: 0.14 s
Wall time: 0.14 s
But I am still unhappy about it, I'd like if the package could import
much faster, because it adds up, when you need to import 7 packages
like that, it's suddenly 1s and that's just too much.
But of course everything within the constrains that Robert has
outlined. From the theoretical point of view, I don't understand why
python cannot just import numpy (or any other package) immediatelly,
and only at the moment the user actually access something, to import
it in real. Mercurial uses a lazy import module, that does exactly
this. Maybe that's an option?
Look into mercurial/demandimport.py.
Use it like this:
In [1]: import demandimport
In [2]: demandimport.enable()
In [3]: %time import numpy
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s
That's pretty good, huh? :)
Unfortunately, numpy cannot work with lazy import (yet):
In [5]: %time from numpy import array
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (17, 0))
---
AttributeErrorTraceback (most recent call last)
[skip]
/usr/lib/python2.5/site-packages/numpy/lib/index_tricks.py in module()
14 import function_base
15 import numpy.core.defmatrix as matrix
--- 16 makemat = matrix.matrix
17
18 # contributed by Stefan van der Walt
/home/ondra/ext/sympy/demandimport.pyc in __getattribute__(self, attr)
73 return object.__getattribute__(self, attr)
74 self._load()
--- 75 return getattr(self._module, attr)
76 def __setattr__(self, attr, val):
77 self._load()
AttributeError: 'module' object has no attribute 'matrix'
BTW, neither can SymPy. However, maybe it shows some possibilities and
maybe it's possible to fix numpy to work with such a lazy import.
On the other hand, I can imagine it can bring a lot more troubles, so
it should probably only be optional.
Ondrej
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Debian: numpy not building _dotblas.so
On Tue, Jul 8, 2008 at 11:48 AM, Tiziano Zito [EMAIL PROTECTED] wrote: Hi numpy-devs, I was the one reporting the original bug about missing ATLAS support in the debian lenny python-numpy package. AFAICT the source python-numpy package in etch (numpy version 1.0.1) does not require atlas to build _dotblas.c, only lapack is needed. If you install the resulting binary package on a system where ATLAS is present, ATLAS libraries are used instead of plain lapack. So basically it was already working before the check for ATLAS was introduced into the numpy building system. Why should ATLAS now be required? It's not as trivial as just reverting that changeset, though. why is that? I mean, it was *working* before... So just removing the two lines from numpy seems to fix the problem in Debian. So far all tests seem to run both on i386 and amd64, both with and without atlas packages installed. And it is indeed faster with the altas packages instaled, yet it doesn't need them to build. I think that's what we want, no? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Debian: numpy not building _dotblas.so
On Tue, Jul 8, 2008 at 9:15 PM, Robert Kern [EMAIL PROTECTED] wrote: On Tue, Jul 8, 2008 at 08:06, Ondrej Certik [EMAIL PROTECTED] wrote: On Tue, Jul 8, 2008 at 11:48 AM, Tiziano Zito [EMAIL PROTECTED] wrote: Hi numpy-devs, I was the one reporting the original bug about missing ATLAS support in the debian lenny python-numpy package. AFAICT the source python-numpy package in etch (numpy version 1.0.1) does not require atlas to build _dotblas.c, only lapack is needed. If you install the resulting binary package on a system where ATLAS is present, ATLAS libraries are used instead of plain lapack. So basically it was already working before the check for ATLAS was introduced into the numpy building system. Why should ATLAS now be required? It's not as trivial as just reverting that changeset, though. why is that? I mean, it was *working* before... So just removing the two lines from numpy seems to fix the problem in Debian. So far all tests seem to run both on i386 and amd64, both with and without atlas packages installed. And it is indeed faster with the altas packages instaled, yet it doesn't need them to build. I think that's what we want, no? Can you give me more details? Sure. :) Was the binary built on a machine with an absent ATLAS? Yes, the binary is always built on a machine with an absent atlas, as the package is build-conflicting with atlas. Show me the output of ldd on _dotblas.so with both ATLAS installed and not. Can you import numpy.core._dotblas explicitly under both? ATLAS installed: [EMAIL PROTECTED]:~/debian$ ldd /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so linux-gate.so.1 = (0xb7fba000) libblas.so.3gf = /usr/lib/atlas/libblas.so.3gf (0xb7c19000) libgfortran.so.3 = /usr/lib/libgfortran.so.3 (0xb7b67000) libm.so.6 = /lib/i686/cmov/libm.so.6 (0xb7b4) libgcc_s.so.1 = /lib/libgcc_s.so.1 (0xb7b33000) libc.so.6 = /lib/i686/cmov/libc.so.6 (0xb79d8000) /lib/ld-linux.so.2 (0xb7fbb000) [EMAIL PROTECTED]:~/debian$ python Python 2.5.2 (r252:60911, Jun 25 2008, 17:58:32) [GCC 4.3.1] on linux2 Type help, copyright, credits or license for more information. import numpy.core._dotblas ATLAS not installed: [EMAIL PROTECTED]:~/debian$ ldd /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so linux-gate.so.1 = (0xb7f2f000) libblas.so.3gf = /usr/lib/libblas.so.3gf (0xb7e82000) libgfortran.so.3 = /usr/lib/libgfortran.so.3 (0xb7dd) libm.so.6 = /lib/i686/cmov/libm.so.6 (0xb7da9000) libgcc_s.so.1 = /lib/libgcc_s.so.1 (0xb7d9c000) libc.so.6 = /lib/i686/cmov/libc.so.6 (0xb7c41000) /lib/ld-linux.so.2 (0xb7f3) [EMAIL PROTECTED]:~/debian$ python Python 2.5.2 (r252:60911, Jun 25 2008, 17:58:32) [GCC 4.3.1] on linux2 Type help, copyright, credits or license for more information. import numpy.core._dotblas Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Debian: numpy not building _dotblas.so
On Tue, Jul 8, 2008 at 10:19 PM, Robert Kern [EMAIL PROTECTED] wrote: On Tue, Jul 8, 2008 at 14:47, Ondrej Certik [EMAIL PROTECTED] wrote: On Tue, Jul 8, 2008 at 9:15 PM, Robert Kern [EMAIL PROTECTED] wrote: On Tue, Jul 8, 2008 at 08:06, Ondrej Certik [EMAIL PROTECTED] wrote: On Tue, Jul 8, 2008 at 11:48 AM, Tiziano Zito [EMAIL PROTECTED] wrote: Hi numpy-devs, I was the one reporting the original bug about missing ATLAS support in the debian lenny python-numpy package. AFAICT the source python-numpy package in etch (numpy version 1.0.1) does not require atlas to build _dotblas.c, only lapack is needed. If you install the resulting binary package on a system where ATLAS is present, ATLAS libraries are used instead of plain lapack. So basically it was already working before the check for ATLAS was introduced into the numpy building system. Why should ATLAS now be required? It's not as trivial as just reverting that changeset, though. why is that? I mean, it was *working* before... So just removing the two lines from numpy seems to fix the problem in Debian. So far all tests seem to run both on i386 and amd64, both with and without atlas packages installed. And it is indeed faster with the altas packages instaled, yet it doesn't need them to build. I think that's what we want, no? Can you give me more details? Sure. :) Was the binary built on a machine with an absent ATLAS? Yes, the binary is always built on a machine with an absent atlas, as the package is build-conflicting with atlas. Show me the output of ldd on _dotblas.so with both ATLAS installed and not. Can you import numpy.core._dotblas explicitly under both? ATLAS installed: [EMAIL PROTECTED]:~/debian$ ldd /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so linux-gate.so.1 = (0xb7fba000) libblas.so.3gf = /usr/lib/atlas/libblas.so.3gf (0xb7c19000) libgfortran.so.3 = /usr/lib/libgfortran.so.3 (0xb7b67000) libm.so.6 = /lib/i686/cmov/libm.so.6 (0xb7b4) libgcc_s.so.1 = /lib/libgcc_s.so.1 (0xb7b33000) libc.so.6 = /lib/i686/cmov/libc.so.6 (0xb79d8000) /lib/ld-linux.so.2 (0xb7fbb000) [EMAIL PROTECTED]:~/debian$ python Python 2.5.2 (r252:60911, Jun 25 2008, 17:58:32) [GCC 4.3.1] on linux2 Type help, copyright, credits or license for more information. import numpy.core._dotblas ATLAS not installed: [EMAIL PROTECTED]:~/debian$ ldd /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so linux-gate.so.1 = (0xb7f2f000) libblas.so.3gf = /usr/lib/libblas.so.3gf (0xb7e82000) libgfortran.so.3 = /usr/lib/libgfortran.so.3 (0xb7dd) libm.so.6 = /lib/i686/cmov/libm.so.6 (0xb7da9000) libgcc_s.so.1 = /lib/libgcc_s.so.1 (0xb7d9c000) libc.so.6 = /lib/i686/cmov/libc.so.6 (0xb7c41000) /lib/ld-linux.so.2 (0xb7f3) [EMAIL PROTECTED]:~/debian$ python Python 2.5.2 (r252:60911, Jun 25 2008, 17:58:32) [GCC 4.3.1] on linux2 Type help, copyright, credits or license for more information. import numpy.core._dotblas Okay, it turns out that libblas on Ubuntu (and I'm guessing Debian) includes the CBLAS interface. $ nm /usr/lib/libblas.a | grep T cblas_ T cblas_caxpy T cblas_ccopy ... This is specific to Debian and its derivatives. Not all libblas's have this. So I stand by my statement that just reverting the change is not acceptable. We need a real check for the CBLAS interface. In the Right. meantime, the Debian package maintainer can patch the file to remove that check during the build for Debian systems. Yes, I just did that. Thanks for the clarification. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Debian: numpy not building _dotblas.so
Hi, we have this problem in Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=489726 The problem is that numpy should not depend on atlas unconditionally, yet it should allow it for users that have it. I am not an expert in blas/lapack/atlas and it's Debian packaging much (I know some people say that atlas packaging in Debian is not very good, actually pretty bad), so I am just forwarding the question here. The problem is with this patch: http://projects.scipy.org/scipy/numpy/changeset/3854 and the question that we have is: doko I'd like to know, if the code was changed to only work with atlas, or if was never working. if it's the latter, then we should use atlas Matthias, Tiziano, feel free to clarify this more. See the above Debian bug for more information and background. Thanks, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] array vs matrix
On Sun, Jun 8, 2008 at 3:54 AM, Anne Archibald [EMAIL PROTECTED] wrote: 2008/6/7 Robert Kern [EMAIL PROTECTED]: On Sat, Jun 7, 2008 at 14:37, Ondrej Certik [EMAIL PROTECTED] wrote: Hi, what is the current plan with array and matrix with regard of calculating sin(A) ? I.e. elementwise vs sin(A) = Q*sin(D)*Q^T? Is the current approach (elementwise for array and Q*sin(D)*Q^T for matrix) the way to go? I don't believe we intend to make numpy.matrix any more featureful. I don't think it's a good idea for you to base sympy.Matrix's capabilities in lock-step with numpy.matrix, though. There are very different constraints at work. Please, do what you think is best for sympy.Matrix. Let me elaborate somewhat: We recently ran across some painful quirks in numpy's handling of matrices, and they spawned massive discussion. As it stands now, there is significant interest in reimplementing the matrix object from scratch, with different behaviour. So emulating its current behaviour is not a win. For consistency, it makes a certain amount of sense to have sin(A) compute a matrix sine, since A**n computes a matrix power. But looking at the matrix exponential, I see that we have several implementations, none of which is satisfactory for all matrices. I would expect the matrix sine to be similar - particularly when faced with complex matrices - so perhaps needing an explicit matrix sine is a good thing? Also worth noting is that this problem can be evaded with namespaces; matrix sin could be scipy.matrixmath.sin, abbreviated perhaps to mm.sin, as opposed to np.sin. I see. Thanks Robert and Anne for the replies. That's all I needed to know. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy 1.1: import numpy fails in Debian
On Mon, Jun 9, 2008 at 6:20 PM, Ondrej Certik [EMAIL PROTECTED] wrote: Hi, I tried to package numpy 1.1 and upload it to Debian, all worked fine, I installed the package and: In [1]: import numpy --- ImportError Traceback (most recent call last) /home/ondra/debian/packages/numpy/ipython console in module() /usr/lib/python2.5/site-packages/numpy/__init__.py in module() 105 import random 106 import ctypeslib -- 107 import ma global ma = undefined 108 109 # Make these accessible from numpy name-space ImportError: No module named ma So I tried to investigate where the problem is, but I didn't figure it out. I am sending the build log. If anyone knows what's wrong, it'd be very helpful. The only thing I figured out so far is that the ma package gets installed into: debian/tmp/usr/lib/python2.5/site-packages/numpy/ma but it is not copied into the directory: debian/python-numpy/usr/share/pyshared/numpy/ from which the package is made. So it's maybe a bug in some Debian package, rather than in numpy itself. But maybe some numpy developers know what is so different on the ma package (which as I understand was added in 1.1). Andrew Straw fixed the problem, so the new numpy package is in incoming: http://incoming.debian.org/ and should be on Debian mirrors tomorrow. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] array vs matrix
Hi, what is the current plan with array and matrix with regard of calculating sin(A) ? I.e. elementwise vs sin(A) = Q*sin(D)*Q^T? Is the current approach (elementwise for array and Q*sin(D)*Q^T for matrix) the way to go? We are solving the same problem in SymPy: http://groups.google.com/group/sympy/browse_thread/thread/6cfe66b012b24948 so I'd like to have the exact same behavior as numpy has, so that we are compatible. But I was wondering if the current status is ok, or if many numpy developers (or users?) would like this to be changed. I think a clean solution is to treat matrices as mathematical objects, i.e. A*B and sin(A) being matrix multiplication and matrix sin respectively, while leaving the array for elementwise operations. But above all, I'd like to be compatible with numpy, otherwise it will be a mess. Thanks for any feedback, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ticket #798: `piecewise` exposes raw memory
On Wed, May 21, 2008 at 11:30 AM, Stéfan van der Walt [EMAIL PROTECTED] wrote:
Referring to
http://scipy.org/scipy/numpy/ticket/798
`piecewise` uses `empty` to allocate output memory. If the conditions
do not sufficiently cover the output, then raw memory is returned,
e.g.,
{{{
import numpy as np
np.piecewise([0,1,2],[True,False,False],[1])
}}}
A patch which addresses the issue is available here for review:
http://codereview.appspot.com/1105
Documentation is being updated on the wiki.
I'd like to invite everyone to take part in the review. It's fun, it's
just talking, no coding. :)
Ondrej
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ticket #798: `piecewise` exposes raw memory
On Wed, May 21, 2008 at 11:53 AM, Ondrej Certik [EMAIL PROTECTED] wrote:
On Wed, May 21, 2008 at 11:30 AM, Stéfan van der Walt [EMAIL PROTECTED]
wrote:
Referring to
http://scipy.org/scipy/numpy/ticket/798
`piecewise` uses `empty` to allocate output memory. If the conditions
do not sufficiently cover the output, then raw memory is returned,
e.g.,
{{{
import numpy as np
np.piecewise([0,1,2],[True,False,False],[1])
}}}
A patch which addresses the issue is available here for review:
http://codereview.appspot.com/1105
Documentation is being updated on the wiki.
I'd like to invite everyone to take part in the review. It's fun, it's
just talking, no coding. :)
Thanks Robert for taking part in the review:
http://codereview.appspot.com/1105/diff/22/122#newcode566
That's the way to go.
Ondrej
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] let's use patch review
On Thu, May 15, 2008 at 3:08 PM, David Huard [EMAIL PROTECTED] wrote: 2008/5/14 David Cournapeau [EMAIL PROTECTED]: On Wed, 2008-05-14 at 13:58 -1000, Eric Firing wrote: What does that mean? How does one know when there is a consensus? There can be a system to make this automatic. For example, the code is never commited directly to svn, but to a gatekeeper, and people vote by an email command to say if they want the patch in; when the total number of votes is above some threshold, the gatekeeper commit the patch. There is about 5 commits/day, I don't think it's a good idea to wait for a vote on each one of them. Me neither. I think just one reviewer is enough. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] let's use patch review
Hi, I read the recent flamebate about unittests, formal procedures for a commit etc. and it was amusing. :) I think Stefan is right about the unit tests. I also think that Travis is right that there is no formal procedure that can assure what we want. I think that a solution is a patch review. Every big/succesful project does it. And the workflow as I see it is this: 1) Travis will fix a bug, and submit it to a patch review. If he is busy, that's the only thing he will do 2) Someone else reviews it. Stefan will be the one who will always point out missing tests. 3) There needs to be a common consensus that the patch is ok to go in. 4) when the patch is reviewed and ok to go in, anyone with a commit access will commit it. I think it's as simple as that. Sometimes no one has enought time to write a proper test, yet someone has a free minute to fix a bug. Then I think it's ok to put the code in, as I think it's good to fix a bug now. However, the issue is definitely not closed and the bug is not fixed (!) until someone writes a proper test. I.e. putting code in that is not tested, however it doesn't break things, is imho ok, as it will not hurt anyone and it will temporarily fix a bug (but of course the code will be broken at some point in the future, if there is no test for it). Now, the problem is that all patch review tools sucks in some way. Currently the most promissing is the one from Guido here: http://code.google.com/p/rietveld/ it's opensource, you can run it on your server, or use it online here: http://codereview.appspot.com/ I suggest you to read the docs how to use it, I am still learning it. Also it works fine for svn, but not for Mercurial, so we are not using it in SymPy yet. So to also do some work besides just talk, I started with this issue: http://projects.scipy.org/scipy/numpy/ticket/788 and submitted the code (not my code though:) in there for a review here: http://codereview.appspot.com/953 and added some comments. So what do you think? Ondrej P.S. efiring, my comments are real questions to your patch. :) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Py3K
On Wed, Apr 16, 2008 at 2:56 PM, Stéfan van der Walt [EMAIL PROTECTED] wrote: On 16/04/2008, Fernando Perez [EMAIL PROTECTED] wrote: On Tue, Apr 15, 2008 at 1:27 PM, Charles R Harris Oh, and making the transition will be made a lot easier by having a complete set of tests. Getting the tests and documentation into good state might be the best focus for our 1.2 effort. +1 Bonus points for whoever adds coverage reporting to our test suite, so that we can get at least basic metrics of coverage by: - docstrings - doctests - plain tests I'd love to see that. In the meantime, here is a figleaf coverage report: http://mentat.za.net/numpy/coverage/ Wow, figleaf is really cool. I didn't know about this tool. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy release
Hi Jarrod, any news with the 1.0.5? If you have same prerelease, I'd like to test it. Debian has just moved from python2.4 to python2.5 yesterday, so I'd like to test numpy in advance, I am sure there will be some issues to fix. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] mercurial now has free hosting too
Hi, since there was so much discussion whether bzr or hg, Mercurial has now free hosting too: http://freehg.org/ also Mercurial 1.0 was finally released yesterday. Bzr has Launchpad, that's one of the (main) reasons ipython is investigating it, so I am still learning how to use bzr, but it's not really so much different. The only little annoying thing I discovered so far is that it feels slower than hg, on regular tasks, like bzr st, bzr pull, bzr up, etc. Only a little bit, but still it creates the feeling in me, that something is missing -- it's like if you are used to a fast car and then you get into a slower car - even though it's fast enough to drive you to the shop, you still are missing something, at least I am. :) But generally I wanted to say, that I think bzr is a good choice too. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy/Cython Google Summer of Code project idea
On Thu, Mar 6, 2008 at 9:48 PM, Robin [EMAIL PROTECTED] wrote: On Thu, Mar 6, 2008 at 6:15 PM, Fernando Perez [EMAIL PROTECTED] wrote: Any student interested in this should quickly respond on the list; such a project would likely be co-mentored by people on the Numpy and Cython teams, since it is likely to require expertise from both ends. Hello, I would like to register my keen interest in applying for Numpy/Scipy GSoC project. I am a first year PhD student in Computational Neuroscience and my undergraduate degree was in Mathematics. I have been using Numpy and Scipy for my PhD work for a few months now and have been building up to trying to contribute something to the project - I am keen to get more substantial real world programming experience... The projects described involving Cython definitely interest me, although I don't yet have a sufficient understanding of the Python C-API and Pyrex/Cython to gauge how demanding they might be. As a PhD student in the UK I don't have any official summer vacation, so I wouldn't be able to work full time on the project (I also have a continuation report due just before the GSoC final deadline which is a bit annoying). However I currently work 20 hours per week part time anyway, so I'm confident that I could replace that with GSoC and still keep up with my studies. Just a note, that the usual commitment is 40 hours/week, i.e. a full time job. See e.g.: http://wiki.python.org/moin/SummerOfCode/Expectations Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy 1:1.0.4: numpy.average() returns the wrong result with weights
On Sun, Feb 24, 2008 at 5:41 AM, Albert Strasheim [EMAIL PROTECTED] wrote: Hello, - Original Message - From: Robert Kern [EMAIL PROTECTED] To: Discussion of Numerical Python numpy-discussion@scipy.org Sent: Saturday, February 23, 2008 3:30 AM Subject: Re: [Numpy-discussion] numpy 1:1.0.4: numpy.average() returns the wrong result with weights Ondrej Certik wrote: I'll add it. I registered on the trac, as required, but I am still denied, when filling my username and password when logging in. How can I create an account? That should have done it. When you say you are denied, exactly what happens? I've run into times when I've logged in and I get the unaltered front page again. Logging in again usually works. There is something strange going on. Logging in on projects.scipy.org/scipy/numpy usually redirects you to scipy.org/scipy/numpy, at which point you need to log in again. I tried it several times now - register - login - it doesn't accept the password. Then it redirects to scipy.org/scipy/numpy as you said. Then I went back to the original page and then the login worked. (But I did the same trick before and it didn't work). Anyway, it seems to be working now, thanks for the help. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy 1:1.0.4: numpy.average() returns the wrong result with weights
Hi, more details in this bug report. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=467095 The bug report offers a fix for this problem. It seems to me this is not fixed even in the latest svn. Thanks, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy 1:1.0.4: numpy.average() returns the wrong result with weights
On Sat, Feb 23, 2008 at 2:10 AM, Travis E. Oliphant [EMAIL PROTECTED] wrote: Ondrej Certik wrote: Hi, more details in this bug report. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=467095 The bug report offers a fix for this problem. It seems to me this is not fixed even in the latest svn. Is there a ticket on the NumPy trac for this? We won't see it if there isn't. Thanks for pointing us to the bug. I'll add it. I registered on the trac, as required, but I am still denied, when filling my username and password when logging in. How can I create an account? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy and C++ integration...
On Feb 5, 2008 11:23 AM, David Cournapeau [EMAIL PROTECTED] wrote: Gael Varoquaux wrote: On Tue, Feb 05, 2008 at 09:15:29AM +0100, Sebastian Haase wrote: Can ctypes do this ? No. Ctypes is only a way of loading C (and not C++) libraries in Python. That makes it very simple, but not very powerful. I would not call ctypes not very powerful :) For sure you cannot do the same way as swig does, but you could imagine some automatic scheme to solve Sebastian's problem. Typically, having a C wrapper automatically generated from the C++ headers, you could use the ctypes code generator, and you have something almost automatic (with a bit some boilerplate code in python, maybe). cheers, Also feel free to extend this wiki: http://wiki.cython.org/WrappingCorCpp I use Cython, mostly for the same reasons Gael is using ctypes - it's trivial. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy and C++ integration...
On Feb 5, 2008 11:52 AM, Gael Varoquaux [EMAIL PROTECTED] wrote: On Tue, Feb 05, 2008 at 11:48:38AM +0100, Ondrej Certik wrote: I use Cython, mostly for the same reasons Gael is using ctypes - it's trivial. Actually, when I want to do something really trivial, I use scipy.weave.inline ( see http://scipy.org/PerformancePython for an example of scipy.weave.inline use). Of course it doesn't work when linking to external libraries, but to accelerate a for loop, it's great. Yep. The power of Cython/Pyrex strategy is that you can work as I described here: http://ondrej.certik.cz/development/ Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Read-only mercurial mirror of numpy trunk available
On Jan 27, 2008 6:57 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: la, 2008-01-19 kello 14:15 +0900, David Cournapeau kirjoitti: Pauli Virtanen wrote: pe, 2008-01-18 kello 18:06 +0900, David Cournapeau kirjoitti: Hi there, I got a mercurial mirror of numpy available. I put some basic info on the wiki http://scipy.org/scipy/numpy/wiki/HgMirror Nice! Don't hesitate to put more info there. Although I would consider myself relatively proficient with bzr, mercurial has some few important differences that I am still not sure to understand (in particular the lack of one branch is one directory concept) I think I don't have edit access to the Trac Wiki... Anyway, it'd be useful to also have either hg or bzr read-only mirror of scipy too! I'd actually like to see these mirrors hosted at some single official place (svn.scipy.org?): If everyone maintains their own SVN mirror, the resulting repositories typically turn out unrelated as far as the SCM programs are concerned (because of different start points, different conversion tools etc.), and one cannot easily pull and merge changes from other people's repositories. hgsvn makes only repository-local svn tags. They go in .hg/localtags, from where they can be copied manually. I don't think hgsvn has an option to put the tags into .hgtags (from where they would be passed on automatically when the repository is cloned). Since tags seem to be handled through plain files in mercurial, would copying localtags into .hgtags work ? It will work by just pasting the contents of localtags to .hgtags and committing. However, this would need to be done periodically and would pollute the history a bit. Better solution is to keep the tags in a Mercurial Queues patch, as I did. See for yourself: http://hg.certik.cz/numpy-hg/ BTW this mirror is much faster for me. You can browse the full history with tags online, you can clone like this: hg clone http://hg.certik.cz/numpy-hg/ It will convert the MQ patch to a regular commit, so you won't see the difference. But I, on the server, will just do: $ hg qpop Patch queue now empty $ hg pull pulling from /home/ondra/scipy/trunk searching for changes no changes found $ hg qpush applying tags.patch Now at: tags.patch Then I'll copy the updated localtags to .hgtags and do: $ hg qrefresh and that's it. No history pollution. If I decide I don't wont't the tags, I just do hg qpop and tags are gone. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to work with mercurial and numpy right now
Hi, if you want to play with Mercurial now (without forcing everyone else to leave svn), I suggest this: http://cheeseshop.python.org/pypi/hgsvn I tried that and it works. It's a very easy way to create a hg mirror at your computer. And then you can take this as the official upstream repository (which you don't have write access to). Whenever somone commits to the svn, you just do hgpullsvn and it updates your mercurial repo. Then you just clone it and create branches, for example the scons branch can be easily managed like this. Then you prepare patches, against your official local mercurial mirror, using for example hg export, or something, those patches should be possible to apply against the svn repository as well. You sent them for review and then (you or someone else) commit them using svn, then you'll hgpullsvn your local mercurial mirror and merge the changes to all your other branches. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Jan 5, 2008 8:15 PM, Fernando Perez [EMAIL PROTECTED] wrote: On Jan 5, 2008 12:08 PM, David M. Cooke [EMAIL PROTECTED] wrote: On Jan 4, 2008, at 13:58 , Fernando Perez wrote: My vote so far is for hg, for performance reasons but also partly because sage and sympy already use it, two projects I'm likely to interact a lot with and that are squarely in line with the ipython/numpy/scipy/matplotlib world. Since they went first and made the choice, I'm happy to let that be a factor in my decision. I'd rather use a tool that others in the same community are also using, especially when the choice is a sound one on technical merit alone. Just my 1e-2... +1 on mercurial. It's what I use these days (previously, I used darcs, which I still like for its patch-handling semantics, but its dependence on Haskell, and the dreaded exponential-time merge are a bit of a pain). Regarding the 'record' capapbilities of darcs which were indeed very nice, here's something that was recently mentioned on the sage list: I noticed that Mercurial 0.9.5 has a record extension that mimics the darcs record functionality of interactively asking what changes you want to commit out of a file. I know there was discussion of this a while ago. Reference: http://www.selenic.com/pipermail/mercurial/2007-October/015150.html under the New extensions heading. See also http://www.selenic.com/mercurial/wiki/index.cgi/RecordExtension Anyways, I'm just posting this as an FYI. It might be nice to expose this functionality to sage, if we haven't already. Thanks, Jason Kirill (a sympy developer) has also sent patches for qrecord (record for mercurial queues) http://www.selenic.com/pipermail/mercurial-devel/2007-December/003953.html Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
Imagine the pain in the other direction, which was my experience :) I actually did not believe at first that it was so bad, and thought I was doing something wrong. At least, it certainly convinced me that SVN was not easier than DVCS. It would made me sick. :) I am not familiar with sympy: you are not using trac at all ? Also, how We use googlecode: http://code.google.com/p/sympy/ it works nice for us. did you convert the svn history ? using the mercurial extension. Kirill submitted some patches, so that also branches are converted and tags too. I like the mercurial's way of showing branches and co; bzr does not have anything like that out of the box (there are separate projects to show sources; there is also launchpad of course, but since it is not open source, I do not even consider it for numpy/scipy). On the other hand, the bzr community is more user-friendly: the tool is easier to use I think, the graphical tools are more advanced, at least from my experience. I never used bzr, so I cannot judge. We were basically only deciding between git and Mercurial, but we chose mercurial, because * we are python guys and Mercurial is in python+C, very nicely written and they accept patches (Kirill, one sympy developer, has posted several already, to implement features he was missing - he used to use darcs before) * Sage uses it For some time, the big problem of bzr was speed. But bzr accomplished quite a lot the last year: the first time I used mercurial, the speed difference was obvious; it is not so true anymore (they 'feel' the same, basically, but I have not used mercurial extensively, at least compared to bzr). So I think it really boils down to the difficulty of the transition, the availability of third party tools (and also the tool used by other projects similar to numpy, as you mentionned). I know that bzr is used by Canonical, but I think socially, you should choose either mercurial or git. Those are imho the most widespread DVCS. As I said, Mercurial being Python+C was very important for us, so that we can easily fix bugs and implement new functionality in mercurial. Also the commands of mercurial are very similar to svn. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Jan 4, 2008 12:56 PM, David Cournapeau [EMAIL PROTECTED] wrote: Neal Becker wrote: There is a mercurial plugin for trac. as well as a bzr one. The problem is more related to performance issues (cheap things in svn are not cheap in DVCS, and vice-versa). For example, the trac-bzr plugin is really slow for timelines (it takes almost one second on a local server !); I have not tried the trac-mercurial one much. We switched from svn to Mercurial in SymPy, I wrote some info here: http://code.google.com/p/sympy/wiki/Mercurial But basically, once you try svn merge and you go through all the pain and then try DVCS (I only use mercurial and git), you never want to come back. Our central repo is here: http://hg.sympy.org/sympy/ and I can just fully recommend. We converted all our svn history to it, so now, I frequently browse the history of sympy (because every clone of the repo has it) if I need to look at something. I never used that with svn, because it's painfully slow. We were basically only deciding between git and Mercurial, but we chose mercurial, because * we are python guys and Mercurial is in python+C, very nicely written and they accept patches (Kirill, one sympy developer, has posted several already, to implement features he was missing - he used to use darcs before) * Sage uses it Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] planet.scipy.org
On Jan 2, 2008 12:45 PM, Adam Mercer [EMAIL PROTECTED] wrote: On Dec 31, 2007 10:43 PM, Jarrod Millman [EMAIL PROTECTED] wrote: Hey, I just wanted to announce that we now have a NumPy/SciPy blog aggregator thanks to Gaël Varoquaux: http://planet.scipy.org/ When I try to load http://planet.scipy.org I get an error saying that the page doesn't exist, however the scipy home page, i.e. http://www.scipy.org, now appears to be the planet aggregator is this just a temporary DNS issue? It works for me. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] planet.scipy.org
On Dec 31, 2007 11:43 PM, Jarrod Millman [EMAIL PROTECTED] wrote: Hey, I just wanted to announce that we now have a NumPy/SciPy blog aggregator thanks to Gaël Varoquaux: http://planet.scipy.org/ Cool! I was going to write an email that it could be a good idea. Thanks Gael. Feel free to contact me if you have a blog that you would like included. Is it just for anything, or just scipy/numpy related? I use this blog: http://ondrejcertik.blogspot.com/ however, since I am new to blogging, I am not sure what the best practise is. I blog about a lot of things (SymPy, Sage, Debian), but I think just something is suitable to be on planet.scipy.org. So I think the best way is to just use tags. So could you please put this into the planet.conf? [http://ondrejcertik.blogspot.com/feeds/posts/default/-/scipy] name = Ond#345;ej #268;ertiacute;k and I'll be tagging scipy related blogposts with the scipy tag (currently 0 posts:). Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] gfortran/g77+f2py vs gcc+Cython speed comparison
Hi John, thanks for the response. This is very surprising indeed. I have just tried to run your code but can't get the fortran generated module to work. But I wonder how you Yes, I really didn't try to make it robust, I just made it work for me on Debian unstable. got such a difference, because in my recent comparisons I generally got about 0-2 % faster (i.e. negligible) results with gfortran compared to gcc and a much bigger difference (in favour of fortran) with the Intel compilers. In fact for simple tests gcc and gfortran produce almost identical assembler code (use -s switch to check). For the more mature compilers (i.e. Intel) fortran is faster as it is inherently easier to optimize (easier data dependency analysis and vectorisation). The main reason I can think of is the gfortran version you are using and the optimisation switches. Any version 4.0.x is seriously suboptimal (first released version). Big gains were made from 4.0-4.1 and 4.1-4.2. I would suggest 4.2 as the minimum version to use. The development version 4.3 is faster still (also for gcc) because of recent vectorisation work. I use: $ gfortran --version GNU Fortran (GCC) 4.2.3 20071123 (prerelease) (Debian 4.2.2-4) Copyright (C) 2007 Free Software Foundation, Inc. GNU Fortran comes with NO WARRANTY, to the extent permitted by law. You may redistribute copies of GNU Fortran under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING $ When using g77 compiler instead of gfortran, I get a speed up 4.8 times. The difference between g77 and gfortran with version 4.3 is now very small, plus gfortran is going to benefit from all the new vectorisation and optimisation work in gcc. It will not be long before it over takes for good. In addition you can use modern fortran 95/2003 code which makes life much easier! In in addition to all that it is very actively maintained and destined to be the default compiler on all linux systems as it is bundled with gcc. Yes, I also think gfortran is the way to go, since g77 is dead upstream. But currently, the g77 still gives quite a lot faster code than gfortran, but I hope it will change. So, what do you think of that? If you are using a modern gfortran version then I suggest you have found a bug, in which case I would take it to the gfortran mailing list. If not, then try version 4.3 which can be downloaded from the gfortran wiki: gcc.gnu.org/wiki/GFortran In the mean time, after Christmas, I'll try and get your code working and see for myself the difference. So your suggestion is to prepare a clean fortran example, that, when compiled with gfortran, runs a lot slower, than if compiled with g77? Yep, I can do that. So what is your personal opinion about fortran vs C for scientific computing? From the social point of view, I much prefer C, since a lot more people know it, but as you say, the good fortran compilers are a lot better, than the C compilers. So I don't know. Probably depends on the project and a code I want to reuse. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] gfortran/g77+f2py vs gcc+Cython speed comparison
Hi,
I needed to write 2D Ising model simulation into my school and I
decided to compare the two possible solutions how to do it, so I of
course wrote
it in Python, then rewrote it in Fortran + f2py, and also Cython. What
is better? Read below. :) But for the impatient, I am going to use
Cython, reasons below.
CCing to Cython, numpy (f2py is discussed there), and sage-devel
(there are people there who could be interested in these kinds of
comparisons).
The code is available at:
http://hg.sharesource.org/isingmodel/
How to play with that - just do this (after installing Mercurial):
$ hg clone http://hg.sharesource.org/isingmodel/
[...]
$ cd isingmodel
$ hg up db7dd01cdc26 # just to be sure
that we are talking about the same revision / code
$ make
[...]
$ time python simulate.py
[...]
real0m2.026s
user0m1.988s
sys 0m0.020s
This runs Cython code. Then apply this patch to run fortran code instead:
$ hg di
diff -r db7dd01cdc26 simulate.py
--- a/simulate.py Sun Dec 23 02:23:30 2007 +0100
+++ b/simulate.py Sun Dec 23 02:24:33 2007 +0100
@@ -31,8 +31,8 @@ def MC(mu = 1, temp = 2, dim = 20, steps
J=1 #coupling constant
k=1 #Boltzman constant
-#from mcising import mc
-from pyising import mc
+from mcising import mc
+#from pyising import mc
B = D1(A)
mc(B, dim, steps, temp, H, mu, J, k)
return D2(B)
And then again (and apply the patch below, otherwise it might not work for you):
$ time python simulate.py
[...]
real0m3.600s
user0m3.528s
sys 0m0.052s
So it's a lot slower.
We are comparing many things here - wrappers, my fortran coding skills
vs Cython C code generation and gcc vs gfortran. So I wrote to numpy
mailinglist. First Travis (the author of numpy) suggested:
My experience with similar kinds of comparisons is that gnu fortran
compilers are not very good, especially on 2-d problems. Try using a
different fortran compiler to see if speeds improve.
Then Pearu (the author of f2py) suggested:
Though the problem is 2D, your implementations are essentially
1D. If you would treat the array A as 2D array (and avoid calling
subroutine p) then you would gain some 7% speed up in Fortran.
When using -DF2PY_REPORT_ATEXIT for f2py then a summary
of timings will be printed out about how much time was spent
in Fortran code and how much in the interface. In the given case
I get (nsteps=5):
Overall time spent in ...
(a) wrapped (Fortran/C) functions : 1962 msec
(b) f2py interface, 60 calls :0 msec
(c) call-back (Python) functions:0 msec
(d) f2py call-back interface, 0 calls :0 msec
(e) wrapped (Fortran/C) functions (acctual) : 1962 msec
that is, most of the time is spent in Fortran function and no time
in wrapper. The conclusion is that the cause of the
difference in timings is not in f2py or cpython generated
interfaces but in Fortran and C codes and/or compilers.
Some idiom used in Fortran code is just slower than in C..
For example, in C code you are doing calculations using
float precision but in Fortran you are forcing double precision.
HTH,
Pearu
PS: Here follows a setup.py file that I used to build the
extension modules instead of the Makefile:
#file: setup.py
def configuration(parent_package='',top_path=None):
from numpy.distutils.misc_util import Configuration
config = Configuration('',parent_package,top_path)
config.add_extension('mcising', sources=['mcising.f'],
define_macros = [('F2PY_REPORT_ATEXIT',1)]
)
#config.add_extension('pyising', sources=['pyising.pyx'])
return config
from numpy.distutils.core import setup
setup(configuration = configuration)
and then quickly added
When using g77 compiler instead of gfortran, I get a speed
up 4.8 times.
Btw, a line in a if statement of the fortran code
should read `A(p(i,j,N)) = - A(p(i,j,N))`.
(btw I have no idea how it could work for me without the A(p(i,j,N)) =
- A(p(i,j,N)) fix, quite embarassing)
So then we discussed it on #sympy IRC:
* Now talking on #sympy
ondrej hi pearu, thanks a lot for testing it!
ondrej 4.8 speedup, jesus christ. so the gfortran sucks a lot
pearu hey ondrej
* ondrej is trying g77
pearu gortran has advantage of being Fortran 90 compiler
ondrej g77 is depricated in Debian and dead upstram (imho)
pearu yes but I guess those guys who are maintaing g77/gfortran,
those do not crunch numbers much;)
pearu g77 has a long development history and is well optimized
ondrej I fear the fortran is a bad investment for new projects
ondrej I know g77 is well optimized, but it's dead
pearu gfortran is a write from scratch and is too young
ondrej do you think many people still write fortran? I use it just
because g77 was faster than gcc
pearu g77 is certainly not dead, scientist use it a lot because of its speed
ondrej btw`A(p(i,j,N)) = - A(p(i,j,N))`.
ondrej means my fortran code was broken
[Numpy-discussion] ising model: f2py vs cython comparison
Hi, I need to write 2D Ising model simulation into my school, so I wrote it in Python, then rewrote it in Fortran + f2py, and also Cython: http://hg.sharesource.org/isingmodel/ And Cython solution is 2x faster than f2py. I understand, that I am comparing many things - wrappers, my fortran coding skills vs Cython C code generation and gcc vs gfortran. But still any ideas how to speed up fortran+f2py solution is very welcomed. How to play with that - just do this (after installing Mercurial): $ hg clone http://hg.sharesource.org/isingmodel/ [...] $ hg up db7dd01cdc26 # just to be sure that we are talking about the same revision / code $ cd isingmodel $ make [...] $ time python simulate.py [...] real0m2.026s user0m1.988s sys 0m0.020s This runs Cython code. Then apply this patch to run fortran code instead: $ hg di diff -r db7dd01cdc26 simulate.py --- a/simulate.py Sun Dec 23 02:23:30 2007 +0100 +++ b/simulate.py Sun Dec 23 02:24:33 2007 +0100 @@ -31,8 +31,8 @@ def MC(mu = 1, temp = 2, dim = 20, steps J=1 #coupling constant k=1 #Boltzman constant -#from mcising import mc -from pyising import mc +from mcising import mc +#from pyising import mc B = D1(A) mc(B, dim, steps, temp, H, mu, J, k) return D2(B) And then again: $ time python simulate.py [...] real0m3.600s user0m3.528s sys 0m0.052s So it's a lot slower. Maybe I did some stupid mistake, like doing different amount of steps in Cython and Fortran, but I tried to check that and if I did everything all right, it's interesting to find why is fortran+f2py slower. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ising model: f2py vs cython comparison
On Dec 23, 2007 2:29 AM, Ondrej Certik [EMAIL PROTECTED] wrote: Hi, I need to write 2D Ising model simulation into my school, so I wrote it in Python, then rewrote it in Fortran + f2py, and also Cython: http://hg.sharesource.org/isingmodel/ And Cython solution is 2x faster than f2py. I understand, that I am comparing many things - wrappers, my fortran coding skills vs Cython C code generation and gcc vs gfortran. But still any ideas how to speed up fortran+f2py solution is very welcomed. How to play with that - just do this (after installing Mercurial): $ hg clone http://hg.sharesource.org/isingmodel/ [...] $ hg up db7dd01cdc26 # just to be sure that we are talking about the same revision / code $ cd isingmodel The last two lines should have been interchanged of course: $ cd isingmodel $ hg up db7dd01cdc26 Also you may want to edit the Makefile: all: f2py --fcompiler=gnu95 --f77flags=-W -Wall -fdefault-real-8 -c -m mcising mcising.f cython pyising.pyx gcc -fPIC -O3 -I/usr/include/python2.4/ -I/usr/share/python-support/python-numpy/numpy/core/include/numpy/ -c -o pyising.o pyising.c gcc -shared pyising.o -o pyising.so Especially the include path to numpy/*.h files. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Fwd: SSE2 segfault
Hi, forwarding more information about the problem + backtrace. Ondrej -- Forwarded message -- From: Jan Medlock [EMAIL PROTECTED] Date: Dec 21, 2007 4:17 AM Subject: Re: Bug#448530: bug reproduced To: Ondrej Certik [EMAIL PROTECTED] Cc: [EMAIL PROTECTED], Kumar Appaiah [EMAIL PROTECTED] On Thu, Dec 20, 2007 at 08:01:01PM +0100, Ondrej Certik wrote: Paul Metcalfe has sent me this explanation: I think it's a packaging problem with linear algebra. ... Hi Ondrej, I'm still here. I saw the bug in the NumPy bug tracker that suggests the problem is specific to Debian Ubuntu and Paul Metcalfe's details about linking. Linking against ATLAS instead of BLAS LAPACK may indeed ultimately be the problem and I'm afraid I don't understand the details. That said, I believe the problem is SSE2 specific. The machine that I'm having the problem with is an Intel Pentium4 with SSE2, but I also have an AMD K7 machine that has 3DNow but not SSE2. The AMD machine does not exhibit the bug with the 3DNow version of ATLAS. (Of course neither do with the SSE or base versions of ATLAS.) I haven't participated in the thread on the NumPy mailing list because I don't see a way to reply to the thread since I wasn't subscribed at the time... Attached is a gdb backtrace. Please let me know how I can make it more informative, if necessary. Thanks again for tracking down this bug, Jan (no debugging symbols found) Using host libthread_db library /lib/i686/cmov/libthread_db.so.1. (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) [Thread debugging using libthread_db enabled] (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) [New Thread 0xb7db18c0 (LWP 23839)] (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb7db18c0 (LWP 23839)] 0xb760afb1 in ATL_dgemvT_a1_x1_b0_y1 () from /usr/lib/sse2/libf77blas.so.3 #0 0xb760afb1 in ATL_dgemvT_a1_x1_b0_y1 () from /usr/lib/sse2/libf77blas.so.3 #1 0xb760363e in ATL_dgemv () from /usr/lib/sse2/libf77blas.so.3 #2 0x0121 in ?? () #3 0x0011 in ?? () #4 0x in ?? () ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] segfault with atlas3-sse2, works with atlas3-sse
On Dec 21, 2007 9:40 AM, Charles R Harris [EMAIL PROTECTED] wrote: On Dec 20, 2007 9:32 AM, Ondrej Certik [EMAIL PROTECTED] wrote: when compiled on Debian, numpy segfaults when used with ATLAS sse2, but works when used against ATLAS sse. More information here: What is the machine on which you are getting the segfault? Is it I don't know which machine the reporter of this bug in Debian uses, but I use Intel Core Duo. possible you are trying to run SSE2 instructions on a machine without SSE2? the /proc/cpuinfo says: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc pni monitor est tm2 xtpr Are you running 32 bits or 64 bits? The Debian versions of ATLAS used to encounter illegal instructions on 64 bit Linux on Intel, but ran fine on 32 bits. You might try compiling your own ATLAS. My processor is imho 64 bits, but I am running 32 bits Debian. Yes, it's probably Debian specific problem, but the solution is to find the bug, and fix it, so that the standard Debian packages work. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] segfault with atlas3-sse2, works with atlas3-sse
Hi, when compiled on Debian, numpy segfaults when used with ATLAS sse2, but works when used against ATLAS sse. More information here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=448530 I have no idea, where the problem is. Do you have any ideas, what we should try? Thanks a lot, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] python-numpy debian package and f2py
Hi, I am a comaintainer of the python-scipy package in Debian and now it seems to be in quite a good shape. However, the python-numpy package is quite a mess, so as it usually goes in opensource, I got fedup and I tried to clean it. But I noticed, that f2py was moved from external package into numpy, however the versions mishmatch: The newest (deprecated) python-f2py package in Debian has a version 2.45.241+1926, so I assume this was the version of f2py, before merging with numpy. However, the f2py in numpy says when executing: Version: 2_3816 numpy Version: 1.0.3 so I assume the version of f2py in numpy is 2_3816? So has the versioning scheme of f2py changed? Another question - since both numpy and f2py is now built from the same source, doesn't f2py simply has the same version as numpy, i.e. 1.0.3? Note: I know there is a newer numpy release, but that's not the point now. I am asking because we probably will have to remove the old python-f2py package and build a new one from the sources of numpy, etc., and it will take some time until this happens (ftpmasters need to remove the old package from the archive, then the new binary package needs to go to the NEW queue for approval etc.), so I would like to make sure I understand the versioning and the future plans with numpy and f2py, before starting the transition in Debian. Actually, does it even make sense to create a python-f2py package? It seems so (to me), it's a separate program. But since you decided to merge it with numpy, what are your thoughts about it? Thanks a lot, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
