Re: [Numpy-discussion] NumPy Governance
On Mon, Dec 5, 2011 at 4:22 AM, Perry Greenfield wrote: > I'm not sure I'm crazy about leaving final decision making for a > board. A board may be a good way of carefully considering the issues, > and it could make it's own recommendation (with a sufficient > majority). But in the end I think one person needs to decide (and that > decision may go against the board consensus, presumably only rarely). > > Why shouldn't that person be you? I haven't contributed to NumPy directly. But I can offer my experience with SymPy. I agree with Perry. Having one person being in charge as the last call (project leader) works excellent in my experience. For SymPy, that person has been me, up until a year ago (when I realized that I am too busy to do a good job as a project leader), when I passed it to Aaron Meurer. We always try to reach consensus, and the project leader's main job is to encourage such discussion. When consensus cannot be reached, he needs to make the decision (that happened maybe once or twice in the last 5 years and it is very rare). There seems to be quite strong "community ownership" in SymPy (that was Stefan's objection). I think the reason being that in fact we probably have something like a "board of members", except that it is informal and it simply consists of people whose opinions the project leader highly values. And I think that it is very easy for anybody who gets involved with SymPy development to become trusted and thus his or her opinion will count. As such, for NumPy I think by default the project leader is Travis, who created it. He became busy in the last few years and so he could appoint a person, who will be the project leader. The list of possible people seems quite simple, I would choose somebody who is involved a lot with NumPy in the last 1 year (let's say): $ git shortlog -ns --since="1 year ago" | head 651 Mark Wiebe 137 Charles Harris 72 David Cournapeau 61 Ralf Gommers 52 rgommers 29 Pearu Peterson 17 Pauli Virtanen 11 Chris Jordan-Squire 11 Matthew Brett 10 Christopher L. Farrow So anybody from the top 5 or 10 people seems ok. This has to be a personal decision, and I don't know what the actual contribution and involvement (and personal ability to be a project leader) is of the above people, so that's why it should be done by Travis (possibly consulting with somebody who he trusts and who is involved). For SymPy, here is the list from the "1 year ago" when I passed the project leadership: $ git shortlog -ns --since="January 2010" --until "January 2011" | head 317 Øyvind Jensen 150 Mateusz Paprocki 93 Aaron Meurer 81 Addison Cugini 79 Brian E. Granger 64 Ronan Lamy 61 Matt Curry 58 Ondřej Čertík 36 Chris Smith 34 Christian Muise It's not exactly accurate, as some of the branches from 2010 were merged in 2011, but it gives you a picture. The above list doesn't tell you who the best person should be. I knew that Aaron would be the best choice, and I consulted it with several "core developers" to see what the "community" thinks, and everybody told me, that if I need to pass it on, Aaron would be the choice. Since this was the first time for me doing this, I simply stated, that Aaron is the project leader from now on. And in couple months we clarified it a little bit, that I am the "owner", in a sense that I own the domain and some servers and other things and I am ultimately responsible for the project (and I still have a say in non-coding related issues, like Google Summer of Code and such). For anything code related, Aaron has the last word, and I will not override it. The precise email is here: https://groups.google.com/d/topic/sympy/i-XD15syvqs/discussion You can compare it to today's list: $ git shortlog -ns --since="1 year ago" | head 805Chris Smith 583 Mateusz Paprocki 508 Aaron Meurer 183 Ronan Lamy 150 Saptarshi Mandal 112 Tom Bachmann 101 Vladimir Perić 93 Gilbert Gede 91 Ondřej Čertík 89 Brian E. Granger So the activity has gone up after I stopped being the bottleneck, and after there was again a clear person, who is in charge and has time for it. Anyway, I just wanted to offer some experience that I gained with SymPy with this regard. As I said, I am not a NumPy developer and as such, this decision should be made by NumPy developers and Travis as the original project leader. I could see a familiar pattern here --- Travis has spent enormous time to develop NumPy and to build a community, and later became busy. This is exactly what happened to me with SymPy (when I was back in Prague, I spent months, every evening, many hours with sympy). In fact, Travis once said at some lecture, that opensource is addictive. And not only that, also, if you develop (start) something, it really feels like it's yours. And then when I didn't have time and I knew I am not doing good job with SymPy, it was probably the hardest decision I had to make to pass
Re: [Numpy-discussion] how to compile Fortran using setup.py
Hi Pearu!
On Sat, Mar 12, 2011 at 2:30 AM, Pearu Peterson
wrote:
>
>
> On Fri, Mar 11, 2011 at 3:58 AM, Ondrej Certik wrote:
>>
>> Hi,
>>
>> I spent about an hour googling and didn't figure this out. Here is my
>> setup.py:
>>
>> setup(
>> name = "libqsnake",
>> cmdclass = {'build_ext': build_ext},
>> version = "0.1",
>> packages = [
>> 'qsnake',
>> 'qsnake.calculators',
>> 'qsnake.calculators.tests',
>> 'qsnake.data',
>> 'qsnake.mesh2d',
>> 'qsnake.tests',
>> ],
>> package_data = {
>> 'qsnake.tests': ['phaml_data/domain.*'],
>> },
>> include_dirs=[numpy.get_include()],
>> ext_modules = [Extension("qsnake.cmesh", [
>> "qsnake/cmesh.pyx",
>> "qsnake/fmesh.f90",
>> ])],
>> description = "Qsnake standard library",
>> license = "BSD",
>> )
>>
>
> You can specify Fortran code, that you don't want to process with f2py, in
> the libraries list
> and then use the corresponding library in the extension, for example:
>
> setup(...
> libraries = [('foo', dict(sources=['qsnake/fmesh.f90']))],
> ext_modules = [Extension("qsnake.cmesh",
> sources =
> ["qsnake/cmesh.pyx"],
> libraries = ['foo']
> )],
> ...
> )
>
> See also scipy/integrate/setup.py that resolves the same issue but just
> using the configuration function approach.
Indeed, I just tried it and it works! Thanks. I am now able to compile
fortran code into the extension module.
The only problem is that I am not able to convince Cython to also
parse the .pyx files, it clashes with the numpy's implementation of
build_ext. Which usually is not a problem, as long as one doesn't need
to compile Fortran and .pyx at the same time.
So cmake seems like the only robust option anyway.
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] rewriting NumPy code in C or C++ or similar
Hi Sturla,
On Tue, Mar 8, 2011 at 6:25 AM, Sturla Molden wrote:
> Den 08.03.2011 05:05, skrev Dan Halbert:
>> Thanks, that's a good suggestion. I have not written Fortran since 1971,
>> but it's come a long way. I was a little worried about the row-major vs
>> column-major issue, but perhaps that can be handled just by remembering
>> to reverse the subscript order between C and Fortran.
>
> In practice this is not a problem. Most numerical libraries for C assume
> Fortran-ordering, even OpenGL assumes Fortran-ordering. People program
> MEX files for Matlab in C all the time. Fortran-ordering is assumed in
> MEX files too.
>
> In ANSI C, array bounds must be known at compile time, so a Fortran
> routine with the interface
>
> subroutine foobar( lda, A )
> integer lda
> double precision A(lda,*)
> end subroutine
>
> will usually be written like
>
> void foobar( int lda, double A[]);
>
> in C, ignoring different calling convention for lda.
>
> Now we would index A(row,col) in Fortran and A[row + col*lda] in C. Is
> that too difficult to remember?
>
> In ANSI C the issue actually only arises with small "array of arrays"
> having static shape, or convoluted contructs like "pointer to an array
> of pointers to arrays". Just avoid those and stay with 1D arrays in C --
> do the 1D to 2D mapping in you mind.
>
> In C99 arrays are allowed to have dynamic size, which mean we can do
>
> void foobar( int lda, double *pA )
> {
> typedef double row_t [lda];
> vector_t *A = (vector_t*)((void*)&pA[0]);
>
> Here we must index A[k][i] to match A(i,k) in Fortran. I still have not
> seen anyone use C99 like this, so I think it is merely theoretical.
>
> Chances are if you know how to do this with C99, you also know how to
> get the subscripts right. If you are afraid to forget "to reverse the
> subscript order between C and Fortran", it just tells me you don't
> really know what you are doing when using C, and should probably use
> something else.
>
> Why not Cython? It has "native support" for NumPy arrays.
>
> Personally I prefer Fortran 95, but that is merely a matter of taste.
+1 to all that you wrote about Fortran. I am pretty much switching to
it from C/C++ for all my numerical work, and then I use Cython to call
it from Python, as well as cmake for the build system.
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to compile Fortran using setup.py
On Fri, Mar 11, 2011 at 12:24 AM, Dag Sverre Seljebotn
wrote:
> On 03/11/2011 09:13 AM, Dag Sverre Seljebotn wrote:
>> On 03/11/2011 07:57 AM, Ondrej Certik wrote:
>>> On Thu, Mar 10, 2011 at 8:25 PM, Robert Kern wrote:
>>>> On Thu, Mar 10, 2011 at 19:58, Ondrej Certik wrote:
>>>>> Hi,
>>>>>
>>>>> I spent about an hour googling and didn't figure this out. Here is my
>>>>> setup.py:
>>>>>
>>>>> setup(
>>>>> name = "libqsnake",
>>>>> cmdclass = {'build_ext': build_ext},
>>>>> version = "0.1",
>>>>> packages = [
>>>>> 'qsnake',
>>>>> 'qsnake.calculators',
>>>>> 'qsnake.calculators.tests',
>>>>> 'qsnake.data',
>>>>> 'qsnake.mesh2d',
>>>>> 'qsnake.tests',
>>>>> ],
>>>>> package_data = {
>>>>> 'qsnake.tests': ['phaml_data/domain.*'],
>>>>> },
>>>>> include_dirs=[numpy.get_include()],
>>>>> ext_modules = [Extension("qsnake.cmesh", [
>>>>> "qsnake/cmesh.pyx",
>>>>> "qsnake/fmesh.f90",
>>>>> ])],
>>>>> description = "Qsnake standard library",
>>>>> license = "BSD",
>>>>> )
>>>>>
>>>>> The qsnake.cmesh extension needs to compile .pyx into .c and later to
>>>>> .o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then
>>>>> link both things. That's it. In other words, this is what I want
>>>>> distutils to do:
>>>>>
>>>>> $ cython cmesh.pyx
>>>>> $ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6
>>>>> -I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include
>>>>> $ gfortran -fPIC -o fmesh.o -c fmesh.f90
>>>>> $ gcc -shared -o cmesh.so cmesh.o fmesh.o
>>>> Difficult if sticking with distutils of any flavor. You would have
>>>> probably to replace the build_ext command to handle both the Fortran
>>>> and the Cython. You may want to give David Cournapeau's Bento a try:
>>>>
>>>> http://pypi.python.org/pypi/bento/
>>> Thanks Robert. I burned most of my today's evening on this, trying to
>>> replace the command, but so far I didn't figure it out. It is indeed
>>> difficult, but I wasn't sure, whether it is because I am not so
>>> familiar with distutils.
>> Kurt Smith spent much more than an evening on Fortran+distutils for the
>> first release of Fwrap, I believe you could find something in the Fwrap
>> 0.1 release...
>>
>> BUT, for Fwrap trunk, Kurt ditched all of that in favour of waf. And it
>> works much, much better (I can simply export "FC=ifort" and it finds the
>> right flags for Intel Fortran). I guess in your case, that means
>> sticking with cmake.
>>
>>> I looked at bento, but I'll simply stick to cmake. I thought that for
>>> a Python package (that uses cython + C or fortran, which I thought is
>>> a pretty standard configuration for scientific computing), I would use
>>> distutils, just like other Python packages.
>> The point of Bento as I understand it is exactly to seperate packaging
>> from building, so that it becomes possible to ditch distutils but still
>> have things behave like a Python package more or less.
>>
>> In theory at least you should be able to plug cmake into Bento (perhaps
>> by having Bento generate files included into your cmake scripts, and
>> then launching a cmake process).
>>
>> David is currently working on using Bento and waf together for building
>> NumPy.
>>
>>> With cmake, I have already figured out how to mix fotran, c, c++,
>>> Cython and Python and everything works nicely together, very robustly.
>>> So if I have to hack distutils anyway, I would write a simple
>>> distutils -> cmake bridge.
>> Please, write a bento+cmake bridge instead :-) Too much time that could
>> have been spent better has gone into distutils already, it's time to
>> just stop using it.
>
> Also, bento comes with a distutils bridge now. So with bento->cmake, one
> should be able to go distutils->bento->cmake.
Thanks Dag for the explanation about bento. Indeed, you are right,
what I want is a solid build system, that can do the job (let's say
cmake in my case), and also create Pythonic packaging, so that it does
what people expect from a Python package.
So the right approach in my case is to stick to distutils to support
setup.py, use bento to hook things in, and write a simple bento->cmake
bridge if I have time.
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to compile Fortran using setup.py
On Thu, Mar 10, 2011 at 8:25 PM, Robert Kern wrote:
> On Thu, Mar 10, 2011 at 19:58, Ondrej Certik wrote:
>> Hi,
>>
>> I spent about an hour googling and didn't figure this out. Here is my
>> setup.py:
>>
>> setup(
>> name = "libqsnake",
>> cmdclass = {'build_ext': build_ext},
>> version = "0.1",
>> packages = [
>> 'qsnake',
>> 'qsnake.calculators',
>> 'qsnake.calculators.tests',
>> 'qsnake.data',
>> 'qsnake.mesh2d',
>> 'qsnake.tests',
>> ],
>> package_data = {
>> 'qsnake.tests': ['phaml_data/domain.*'],
>> },
>> include_dirs=[numpy.get_include()],
>> ext_modules = [Extension("qsnake.cmesh", [
>> "qsnake/cmesh.pyx",
>> "qsnake/fmesh.f90",
>> ])],
>> description = "Qsnake standard library",
>> license = "BSD",
>> )
>>
>> The qsnake.cmesh extension needs to compile .pyx into .c and later to
>> .o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then
>> link both things. That's it. In other words, this is what I want
>> distutils to do:
>>
>> $ cython cmesh.pyx
>> $ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6
>> -I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include
>> $ gfortran -fPIC -o fmesh.o -c fmesh.f90
>> $ gcc -shared -o cmesh.so cmesh.o fmesh.o
>
> Difficult if sticking with distutils of any flavor. You would have
> probably to replace the build_ext command to handle both the Fortran
> and the Cython. You may want to give David Cournapeau's Bento a try:
>
> http://pypi.python.org/pypi/bento/
Thanks Robert. I burned most of my today's evening on this, trying to
replace the command, but so far I didn't figure it out. It is indeed
difficult, but I wasn't sure, whether it is because I am not so
familiar with distutils.
I looked at bento, but I'll simply stick to cmake. I thought that for
a Python package (that uses cython + C or fortran, which I thought is
a pretty standard configuration for scientific computing), I would use
distutils, just like other Python packages.
With cmake, I have already figured out how to mix fotran, c, c++,
Cython and Python and everything works nicely together, very robustly.
So if I have to hack distutils anyway, I would write a simple
distutils -> cmake bridge.
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to compile Fortran using setup.py
Hi,
I spent about an hour googling and didn't figure this out. Here is my setup.py:
setup(
name = "libqsnake",
cmdclass = {'build_ext': build_ext},
version = "0.1",
packages = [
'qsnake',
'qsnake.calculators',
'qsnake.calculators.tests',
'qsnake.data',
'qsnake.mesh2d',
'qsnake.tests',
],
package_data = {
'qsnake.tests': ['phaml_data/domain.*'],
},
include_dirs=[numpy.get_include()],
ext_modules = [Extension("qsnake.cmesh", [
"qsnake/cmesh.pyx",
"qsnake/fmesh.f90",
])],
description = "Qsnake standard library",
license = "BSD",
)
The qsnake.cmesh extension needs to compile .pyx into .c and later to
.o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then
link both things. That's it. In other words, this is what I want
distutils to do:
$ cython cmesh.pyx
$ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6
-I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include
$ gfortran -fPIC -o fmesh.o -c fmesh.f90
$ gcc -shared -o cmesh.so cmesh.o fmesh.o
If I import the relevant commands above by:
from numpy.distutils.core import setup
from numpy.distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy
then it gives me:
running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands
--compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands
--fcompiler options
running build_src
build_src
building extension "qsnake.cmesh" sources
f2py options: []
f2py:> build/src.linux-x86_64-2.6/qsnake/cmeshmodule.c
creating build
creating build/src.linux-x86_64-2.6
creating build/src.linux-x86_64-2.6/qsnake
Reading fortran codes...
Reading file 'qsnake/fmesh.f90' (format:free)
Traceback (most recent call last):
File "setup.py", line 29, in
license = "BSD",
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/core.py",
line 186, in setup
return old_setup(**new_attr)
File "/home/certik1/repos/qsnake/local/lib/python/distutils/core.py",
line 152, in setup
dist.run_commands()
File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py",
line 975, in run_commands
self.run_command(cmd)
File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py",
line 995, in run_command
cmd_obj.run()
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/install.py",
line 55, in run
r = old_install.run(self)
File
"/home/certik1/repos/qsnake/local/lib/python/distutils/command/install.py",
line 577, in run
self.run_command('build')
File "/home/certik1/repos/qsnake/local/lib/python/distutils/cmd.py",
line 333, in run_command
self.distribution.run_command(command)
File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py",
line 995, in run_command
cmd_obj.run()
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build.py",
line 37, in run
old_build.run(self)
File "/home/certik1/repos/qsnake/local/lib/python/distutils/command/build.py",
line 134, in run
self.run_command(cmd_name)
File "/home/certik1/repos/qsnake/local/lib/python/distutils/cmd.py",
line 333, in run_command
self.distribution.run_command(command)
File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py",
line 995, in run_command
cmd_obj.run()
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py",
line 152, in run
self.build_sources()
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py",
line 169, in build_sources
self.build_extension_sources(ext)
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py",
line 334, in build_extension_sources
sources = self.f2py_sources(sources, ext)
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py",
line 593, in f2py_sources
['-m',ext_name]+f_sources)
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/f2py2e.py",
line 342, in run_main
postlist=callcrackfortran(files,options)
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/f2py2e.py",
line 276, in callcrackfortran
postlist=crackfortran.crackfortran(files)
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py",
line 2683, in crackfortran
readfortrancode(files,crackline)
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py",
line 420, in readfortrancode
dowithline(finalline)
File
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py",
line 624, in crackline
analyzelin
Re: [Numpy-discussion] Trac unaware of github move
On Sun, Nov 7, 2010 at 7:17 AM, Ralf Gommers wrote: > On Thu, Nov 4, 2010 at 10:06 PM, Pauli Virtanen wrote: >> Thu, 04 Nov 2010 22:00:47 +0800, Ralf Gommers wrote: >>> I just noticed that the Trac wiki is not displaying updates to files >>> kept in the source tree, for example >>> http://projects.scipy.org/numpy/wiki/TestingGuidelines is stuck at an >>> older version. >>> >>> Can one of the admins point the ReST plugin to github? >> >> That would require more work on the Trac-Git integration front: >> >> http://github.com/pv/githubsimple-trac >> >> It might be more cost-effective to just use links to the Github web >> interface. > > That will require renaming those files in the source tree from *.txt > to *.rst, otherwise there's no way to have github render them > properly. Unless I missed something. Would that be fine? I use .rst instead of .txt for my projects, so that github can render it properly. I don't know if that's the right solution, but it gets the job done. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help with the tensordot function?
On Tue, Sep 7, 2010 at 12:23 PM, Ondrej Certik wrote: > Hi Rick! > > On Fri, Sep 3, 2010 at 4:02 AM, Rick Muller wrote: >> Can someone help me replace a slow expression with a faster one based on >> tensordot? I've read the documentation and I'm still confused. >> >> I have two matrices b and d. b is n x m and d is m x m. I want to replace >> the expression >> >>>>> bdb = zeros(n,'d') >>>>> for i in xrange(n): >>>>> bdb[i,:] = dot(b[i,:],dot(d,b[i,:]) > > I am first trying to reproduce this --- the above is missing one ")" > and also dot() seems to produce a number, but you are assigning it to > bdb[i,:], also you declare bdb as a 1D array. So I tried this: > > http://gist.github.com/568879/ > >> >> with something that doesn't have the for loop and thus is a bit faster. >> >> The first step is >> >>>>> bd = dot(b,d) >> >> However, following this with >> >>>>> bdb = dot(bd,b.T) >> >> doesn't work, since it yields a n x n matrix instead of an n x 1 vector. >> Reading the notes on tensordot makes me think it's the function to use, but >> I'm having trouble grokking the axes argument. Can anyone help? > > In the above gist, I did the following: > > > bd = dot(b, d) > bdb = diag(dot(bd, b.T)) > print bdb > > > which printed the same as: > > > for i in xrange(n): > bdb[i] = dot(b[i,:], dot(d, b[i, :])) > print bdb > > > but I think that this is not what you want, is it? I think you want to > do something like: > > b * d * b.T > > but since b is a (n, m) matrix, the result is a matrix, not a vector, right? Ah, I just noticed you got this resolved in the other thread. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help with the tensordot function?
Hi Rick! On Fri, Sep 3, 2010 at 4:02 AM, Rick Muller wrote: > Can someone help me replace a slow expression with a faster one based on > tensordot? I've read the documentation and I'm still confused. > > I have two matrices b and d. b is n x m and d is m x m. I want to replace > the expression > bdb = zeros(n,'d') for i in xrange(n): bdb[i,:] = dot(b[i,:],dot(d,b[i,:]) I am first trying to reproduce this --- the above is missing one ")" and also dot() seems to produce a number, but you are assigning it to bdb[i,:], also you declare bdb as a 1D array. So I tried this: http://gist.github.com/568879/ > > with something that doesn't have the for loop and thus is a bit faster. > > The first step is > bd = dot(b,d) > > However, following this with > bdb = dot(bd,b.T) > > doesn't work, since it yields a n x n matrix instead of an n x 1 vector. > Reading the notes on tensordot makes me think it's the function to use, but > I'm having trouble grokking the axes argument. Can anyone help? In the above gist, I did the following: bd = dot(b, d) bdb = diag(dot(bd, b.T)) print bdb which printed the same as: for i in xrange(n): bdb[i] = dot(b[i,:], dot(d, b[i, :])) print bdb but I think that this is not what you want, is it? I think you want to do something like: b * d * b.T but since b is a (n, m) matrix, the result is a matrix, not a vector, right? Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] update on the transition to git/github
On Tue, Jul 13, 2010 at 7:50 PM, Joshua Holbrook wrote: > This is awesome! I love github. I really wanted to champion for its > use at the BoF but unfortunately missed it. Indeed, that's awesome. I should say finally! :) Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] comparing floating point numbers
Hi Keith! On Mon, Jul 19, 2010 at 6:40 PM, Keith Goodman wrote: > On Mon, Jul 19, 2010 at 6:31 PM, Ondrej Certik wrote: >> Hi, >> >> I was always using something like >> >> abs(x-y) < eps >> >> or >> >> (abs(x-y) < eps).all() >> >> but today I needed to also make sure this works for larger numbers, >> where I need to compare relative errors, so I found this: >> >> http://www.cygnus-software.com/papers/comparingfloats/comparingfloats.htm >> >> and wrote this: >> >> def feq(a, b, max_relative_error=1e-12, max_absolute_error=1e-12): >> a = float(a) >> b = float(b) >> # if the numbers are close enough (absolutely), then they are equal >> if abs(a-b) < max_absolute_error: >> return True >> # if not, they can still be equal if their relative error is small >> if abs(b) > abs(a): >> relative_error = abs((a-b)/b) >> else: >> relative_error = abs((a-b)/a) >> return relative_error <= max_relative_error >> >> >> Is there any function in numpy, that implements this? Or maybe even >> the better, integer based version, as referenced in the link above? >> >> I need this in tests, where I calculate something on some mesh, then >> compare to the correct solution projected on some other mesh, so I >> have to deal with accuracy issues. > > Is allclose close enough? > > np.allclose(a, b, rtol=1.0001e-05, atol=1e-08) > > Returns True if two arrays are element-wise equal within a tolerance. > > The tolerance values are positive, typically very small numbers. The > relative difference (`rtol` * abs(`b`)) and the absolute difference > `atol` are added together to compare against the absolute difference > between `a` and `b`. thanks for this. This should do the job. I'll give it a shot and report back. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] comparing floating point numbers
Hi, I was always using something like abs(x-y) < eps or (abs(x-y) < eps).all() but today I needed to also make sure this works for larger numbers, where I need to compare relative errors, so I found this: http://www.cygnus-software.com/papers/comparingfloats/comparingfloats.htm and wrote this: def feq(a, b, max_relative_error=1e-12, max_absolute_error=1e-12): a = float(a) b = float(b) # if the numbers are close enough (absolutely), then they are equal if abs(a-b) < max_absolute_error: return True # if not, they can still be equal if their relative error is small if abs(b) > abs(a): relative_error = abs((a-b)/b) else: relative_error = abs((a-b)/a) return relative_error <= max_relative_error Is there any function in numpy, that implements this? Or maybe even the better, integer based version, as referenced in the link above? I need this in tests, where I calculate something on some mesh, then compare to the correct solution projected on some other mesh, so I have to deal with accuracy issues. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_SimpleNewFromData segfaults
On Mon, Oct 5, 2009 at 9:42 PM, Robert Kern wrote: > On Mon, Oct 5, 2009 at 23:34, Ondrej Certik wrote: > >> The only mention of the _import_array() in the documentation that I >> found is here: >> >> http://docs.scipy.org/doc/numpy/reference/c-api.array.html#NO_IMPORT_ARRAY >> >> >> but I don't understand what it means do I have to just call >> _import_array() and then I can use numpy CAPI, or do I also have to >> define those PY_ARRAY_UNIQUE_SYMBOL etc? > > Not _import_array() but import_array(). > > http://docs.scipy.org/doc/numpy/reference/c-api.array.html#importing-the-api > > You don't have multiple files, so you only use import_array() and not > PY_ARRAY_UNIQUE_SYMBOL or NO_IMPORT_ARRAY. > > I'm not really sure what is unclear about the text except that you > searched for the wrong spelling and found the wrong entry. Ah, that's the way. I was using _import_array() and that worked, so I changed that to import_array() and call it just once at the top of the .pyx file and now everything works very nice. Indeed, it is well documented in there, I didn't realize it written one paragraph above it. Thanks for help, all is fine now. Here is how to use that new code in hermes1d from C++: http://groups.google.com/group/hermes1d/msg/54f90f1aa740e93f one can now easily decide if to copy or not to copy the data when constructing the numpy arrays. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_SimpleNewFromData segfaults
On Mon, Oct 5, 2009 at 9:25 PM, Ondrej Certik wrote: > On Mon, Oct 5, 2009 at 8:38 PM, Ondrej Certik wrote: >> On Mon, Oct 5, 2009 at 7:34 PM, Charles R Harris >> wrote: >>> >>> >>> On Mon, Oct 5, 2009 at 7:40 PM, Ondrej Certik wrote: >> [...] >>>> still alive >>>> Segmentation fault >>>> >>>> >>>> >>>> What puzzles me is that there is no debugging print statement just >>>> before the segfault. >>> >>> Maybe you need to flush the buffer. That is a good thing to do when >>> segfaults are about. >> >> I tried to put "fflush(NULL);" after it, but it didn't help. I have >> created a super simple demo for anyone to play: >> >> >> $ git clone git://github.com/certik/segfault.git >> $ cd segfault/ >> $ vim Makefile # <-- edit the python and numpy include paths >> $ make >> $ python test.py >> I am still alive >> Segmentation fault >> >> where test.py is: >> >> $ cat test.py >> import _hermes1d >> v = _hermes1d.test() >> print v >> >> >> and _hermes1d.pyx is: >> >> $ cat _hermes1d.pyx >> def test(): >> cdef npy_intp size >> cdef ndarray newarr >> cdef double *arrsource >> >> size = 10 >> arrsource = malloc(sizeof(double) * size) >> print "I am still alive" >> newarr = PyArray_SimpleNewFromData(1, &size, NPY_DOUBLE, > *>arrsource) >> print "I am dead." >> >> return newarr >> >> >> So I bet there is something very stupid that I am missing. Still >> investigating... > > I didn't call _import_array() ! > > This patch fixes it: > > > diff --git a/_hermes1d.pxd b/_hermes1d.pxd > index 9994c28..f5e8868 100644 > --- a/_hermes1d.pxd > +++ b/_hermes1d.pxd > @@ -54,6 +54,8 @@ cdef extern from "arrayobject.h": > object PyArray_SimpleNewFromData(int nd, npy_intp* dims, int typenum, > void* data) > > + void _import_array() > + > cdef extern from "Python.h": > ctypedef void PyObject > void Py_INCREF(PyObject *x) > diff --git a/_hermes1d.pyx b/_hermes1d.pyx > index e542ddc..7a4beec 100644 > --- a/_hermes1d.pyx > +++ b/_hermes1d.pyx > @@ -2,6 +2,7 @@ def test(): > cdef npy_intp size > cdef ndarray newarr > cdef double *arrsource > + _import_array() > > size = 10 > arrsource = malloc(sizeof(double) * size) > > > > > I think I learned something today the hard way. The only mention of the _import_array() in the documentation that I found is here: http://docs.scipy.org/doc/numpy/reference/c-api.array.html#NO_IMPORT_ARRAY but I don't understand what it means do I have to just call _import_array() and then I can use numpy CAPI, or do I also have to define those PY_ARRAY_UNIQUE_SYMBOL etc? Btw, to explain my original post for future readers --- the real problem was that PyArray_Type was NULL and thus &PyArray_Type segfaulted. That happened in the definition: #define PyArray_SimpleNewFromData(nd, dims, typenum, data)\ PyArray_New(&PyArray_Type, nd, dims, typenum, NULL, \ data, 0, NPY_CARRAY, NULL) so it is *extremely* confusing, since PyArray_SimpleNewFromData() was being called from my code, but PyArray_New never started, it segfaulted in between. I think now it is clear what is going on. Only I don't understand the intention, but I can now get my job done. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_SimpleNewFromData segfaults
On Mon, Oct 5, 2009 at 8:38 PM, Ondrej Certik wrote: > On Mon, Oct 5, 2009 at 7:34 PM, Charles R Harris > wrote: >> >> >> On Mon, Oct 5, 2009 at 7:40 PM, Ondrej Certik wrote: > [...] >>> still alive >>> Segmentation fault >>> >>> >>> >>> What puzzles me is that there is no debugging print statement just >>> before the segfault. >> >> Maybe you need to flush the buffer. That is a good thing to do when >> segfaults are about. > > I tried to put "fflush(NULL);" after it, but it didn't help. I have > created a super simple demo for anyone to play: > > > $ git clone git://github.com/certik/segfault.git > $ cd segfault/ > $ vim Makefile # <-- edit the python and numpy include paths > $ make > $ python test.py > I am still alive > Segmentation fault > > where test.py is: > > $ cat test.py > import _hermes1d > v = _hermes1d.test() > print v > > > and _hermes1d.pyx is: > > $ cat _hermes1d.pyx > def test(): > cdef npy_intp size > cdef ndarray newarr > cdef double *arrsource > > size = 10 > arrsource = malloc(sizeof(double) * size) > print "I am still alive" > newarr = PyArray_SimpleNewFromData(1, &size, NPY_DOUBLE, arrsource) > print "I am dead." > > return newarr > > > So I bet there is something very stupid that I am missing. Still > investigating... I didn't call _import_array() ! This patch fixes it: diff --git a/_hermes1d.pxd b/_hermes1d.pxd index 9994c28..f5e8868 100644 --- a/_hermes1d.pxd +++ b/_hermes1d.pxd @@ -54,6 +54,8 @@ cdef extern from "arrayobject.h": object PyArray_SimpleNewFromData(int nd, npy_intp* dims, int typenum, void* data) +void _import_array() + cdef extern from "Python.h": ctypedef void PyObject void Py_INCREF(PyObject *x) diff --git a/_hermes1d.pyx b/_hermes1d.pyx index e542ddc..7a4beec 100644 --- a/_hermes1d.pyx +++ b/_hermes1d.pyx @@ -2,6 +2,7 @@ def test(): cdef npy_intp size cdef ndarray newarr cdef double *arrsource +_import_array() size = 10 arrsource = malloc(sizeof(double) * size) I think I learned something today the hard way. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyArray_SimpleNewFromData segfaults
On Mon, Oct 5, 2009 at 7:34 PM, Charles R Harris wrote: > > > On Mon, Oct 5, 2009 at 7:40 PM, Ondrej Certik wrote: [...] >> still alive >> Segmentation fault >> >> >> >> What puzzles me is that there is no debugging print statement just >> before the segfault. > > Maybe you need to flush the buffer. That is a good thing to do when > segfaults are about. I tried to put "fflush(NULL);" after it, but it didn't help. I have created a super simple demo for anyone to play: $ git clone git://github.com/certik/segfault.git $ cd segfault/ $ vim Makefile # <-- edit the python and numpy include paths $ make $ python test.py I am still alive Segmentation fault where test.py is: $ cat test.py import _hermes1d v = _hermes1d.test() print v and _hermes1d.pyx is: $ cat _hermes1d.pyx def test(): cdef npy_intp size cdef ndarray newarr cdef double *arrsource size = 10 arrsource = malloc(sizeof(double) * size) print "I am still alive" newarr = PyArray_SimpleNewFromData(1, &size, NPY_DOUBLE, arrsource) print "I am dead." return newarr So I bet there is something very stupid that I am missing. Still investigating... Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] PyArray_SimpleNewFromData segfaults
Hi,
I am getting a segfault in PyArray_SimpleNewFromData in Cython. I am
trying to debug it for the last 4 hours, but still absolutely no clue,
so I am posting it here, maybe someone knows where the problem is:
cdef ndarray array_double_c2numpy(double *A, int len):
from numpy import empty
print "got len:", len
cdef npy_intp dims[10]
cdef double X[500]
print "1"
dims[0] = 3
print "2x"
print dims[0], len
print X[0], X[1], X[2]
cdef npy_intp size
cdef ndarray newarr
cdef double *arrsource
size = 10
arrsource = malloc(sizeof(double) * size)
print "still alive"
newarr = PyArray_SimpleNewFromData(1, &size, 12,
arrsource)
print "I am already dead. :("
print "3"
return empty([len])
Essential is just the line:
newarr = PyArray_SimpleNewFromData(1, &size, 12,
arrsource)
Then I removed all numpy from my computer, downloaded the latest git
repository from:
http://projects.scipy.org/git/numpy.git
applied the following patch:
diff --git a/numpy/core/src/multiarray/ctors.c b/numpy/core/src/multiarray/ctors
index 3fdded0..777563c 100644
--- a/numpy/core/src/multiarray/ctors.c
+++ b/numpy/core/src/multiarray/ctors.c
@@ -1318,6 +1318,7 @@ PyArray_NewFromDescr(PyTypeObject *subtype, PyArray_Descr
intp *dims, intp *strides, void *data,
int flags, PyObject *obj)
{
+printf("entering PyArray_NewFromDescr\n");
PyArrayObject *self;
int i;
size_t sd;
@@ -1553,6 +1554,7 @@ PyArray_New(PyTypeObject *subtype, int nd, intp *dims, int
{
PyArray_Descr *descr;
PyObject *new;
+printf("entering PyArray_New, still kicking\n");
descr = PyArray_DescrFromType(type_num);
if (descr == NULL) {
then installed with:
python setup.py install --home=~/usr
and run my cython program. Here is the output:
$ ./schroedinger
---
This is Hermes1D - a free ODE solver
based on the hp-FEM and Newton's method,
developed by the hp-FEM group at UNR
and distributed under the BSD license.
For more details visit http://hpfem.org/.
---
Importing hermes1d
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_New, still kicking
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_New, still kicking
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
entering PyArray_NewFromDescr
Python initialized
got len: 39601
1
2x
3 39601
0.0 0.0 0.0
still alive
Segmentation fault
What puzzles me is that there is no debugging print statement just
before the segfault. So like if the PyArray_New was not being called.
But looking into numpy/core/include/numpy/ndarrayobject.h, line 1359:
#define PyArray_SimpleNewFromData(nd, dims, typenum, data)\
PyArray_New(&PyArray_Type, nd, dims, typenum, NULL, \
data, 0, NPY_CARRAY, NULL)
It should be called. Does it segfault in the printf() statement above?
Hm. I also tried gdb, but it doesn't step into
PyArray_SimpleNewFromData (in the C file), not sure why.
So both print statements and gdb failed to bring me to the cause,
pretty sad day for me. I am going home now and start with a fresh
head, it just can't segfault like this... I guess I'll start by
creating a simple cython project to reproduce it (the schroedinger
code above is quite involved, it starts a python interpreter inside a
C++ program, etc. etc.).
Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
2009/4/13 Stéfan van der Walt : > 2009/4/13 Eric Firing : >> >> Stéfan, or other git-users, >> >> One feature of hg that I use frequently is "hg serve", the builtin web >> server. I use it for two purposes: for temporary local publishing >> (e.g., in place of using ssh--sometimes it is quicker and easier), and >> for providing access to the very nice hgwebdir.cgi browsing capability >> on local repos. I have looked through git commands etc., and have not >> found an equivalent; am I missing something? The browsing capability of >> hgwebdir.cgi is much better than any gui interface I have seen for git >> or for hg. I realize there is a gitweb.cgi, but having that cgi is not >> the same as being able to publish locally with a single command, and >> then browse. > > The command is > > git instaweb --httpd=webrick Ah, that's nice, thanks for sharing it. I didn't know about it. Works fine for me. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Sun, Apr 12, 2009 at 7:13 PM, David Cournapeau wrote: > Eric Firing wrote: >> It probably does--it is written in python. >> > > Yes, but it is just a script to call git-daemon. I am somewhat doubtful > that it would work on windows, but I would love being proven wrong :) It uses os.fork() which doesn't work on windows. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Sat, Apr 11, 2009 at 8:28 PM, Fernando Perez wrote: > In bzr (and as far as I see, also in hg), this kind of history > rewriting is near impossible, so the best you can do is make a merge > commit and leave all that history in there, visible in the 'second > level' log (indented in the text view). As a project grows many > developers, having all this history merged back into the main project > tree gets unwieldy. Great email. Exactly, that's my experience with hg, it makes it nearly impossible, without misusing mercurial queues. I am just surprised, you have exactly the same experience with bzr, I thought it's only hg. As an amusing story, my roommate here at UNR also switched from svn to hg and was telling me, you know, git is too unfriendly, hg does everything I ever need. I was telling myself, well, after you really start using it, you'll maybe switch to git. Then after couple months he once told me -- "I had to rewrite history, so I tried MQ and that's a disaster, the interface to it is horrible. So I switched to git." I was thinking about Fernando just yesterday, asking myself, hm, I wonder how he is satisfied with bzr. So this email really made me laugh. :) > As for what numpy/scipy should do, I'll leave that to those who > actually contribute to the projects :) I just hope that this view is Exactly, this decision should be done by numpy developers, not me either. > useful to anyone who wants to think about the problem from an angle > different to that of specific commands, benchmarks or features :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Sat, Apr 11, 2009 at 2:12 PM, Neal Becker wrote: > > Why not try asking on mercur...@selenic.com (gmane.comp.version- > control.mercurial.general) Done: http://www.selenic.com/pipermail/mercurial/2009-April/025131.html O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Sat, Apr 11, 2009 at 11:44 AM, Neal Becker wrote: > Ondrej Certik wrote: > >> On Fri, Apr 10, 2009 at 12:43 AM, Eric Firing wrote: >>> Ondrej Certik wrote: >>>> On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau >>>> wrote: >>>>> Ondrej Certik wrote: >>>>>> It is maybe easier to learn how to work with different clones, but >>>>>> once you start working with lots of patches and you need to reclone >>>>>> all the time, then it's the wrong approach to work, as it takes lots >>>>>> of time to copy the whole repository on the disk. >>> >>> This is simply wrong. Mercurial uses hard links for cloning a repo that >> >> On my laptop, recloning the whole repository (with hardlinks) takes >> considerably longer than creating a new branch in the same directory, >> that's a pure fact. > > You can clone a repo using: > cp -al old new > > That should be very fast. > > As long as you then use an editor that behaves correctly with hard links. > If you use emacs you can configure this behavior. Ok, this seems to be pretty fast: $ time cp -al sympy-git.hg/ new real0m0.129s user0m0.020s sys 0m0.084s e.g. this was the mercurial repo. Creating a new branch in git: $ time git co -b new2 Switched to a new branch "new" real0m0.048s user0m0.020s sys 0m0.016s is faster, but I agree, that 0.1s is not an issue for me. Is this going to work on windows? I thought windows don't have hardlinks. In any case, I would prefer to use standard mercurial tools for the job, so if I do: $ time hg clone sympy-git.hg new updating working directory 566 files updated, 0 files merged, 0 files removed, 0 files unresolved real0m1.156s user0m0.948s sys 0m0.164s it still takes over a second. That's too much for me, as this operation of creating new branches is something that I do almost all the time. The way out is to use branches in on repository, and imho that's the correct way, both in git and in mercurial. However, is here anyone who actually uses branches in mercurial? If so, try this: hg clone http://hg.sympy.org/sympy-git.hg cd sympy-git.hg hg branch new2 vim README # do some changes hg ci hg up -C default hg vi and the hg vi doesn't even show your branch names and which branch contains what. let's edit README in the default branch: vim README hg ci now if you do: hg vi it shows the new2 branch, and it shows the main branch diverged, so it doesn't look as nice as in gitk, but it is possible to use. Now let's try mercurial rebase: $ hg up -C new2 $ hg rebase -d default merging README saving bundle to /tmp/ab/hg/sympy-git.hg/.hg/strip-backup/536215160a1c-temp adding branch adding changesets adding manifests adding file changes added 2 changesets with 2 changes to 1 files abort: 00changelo...@536215160a1c: no node! Oops, something went wrong. But commits are still there, so I guess I can safely ignore the error message (could someone clarify?). Now let's say I would like to merge the top two patches, since they both modify readme and I would like to only send one patch. In git, I just do "git rebase -i HEAD~2" tell it in vim which patches to squash and I am done. In mercurial, it's a hell of a job: $ hg qimport -r tip $ hg qimport -r qparent $ hg qpop now at: 2146.diff $ hg qfold 2147.diff $ hg qdelete -r qbase:qtip And I am still not done! I forgot to change the log (git asks you this automatically during the rebase), so we need to import the patch to MQ again: $ hg qimport -r tip $ hg qrefresh -e $ hg qdelete -r qbase:qtip And I am finally done. Now let's say some of the patches in MQ didn't apply after changing the order or some other changes. Then I am in deep troubles, because "hg qpush" fails and I need to modify the patch by hand (that really sucks!). With git, you only use rebase, and rebase is pretty powerful tool that can handle most of the conflicts itself, and if it can't, it asks you to resolve it, I assume just like mercurial rebase, but unfortunately mercurial rebase can't be used to mangle patches or history. I would like to ask mercurial proponents on this list to please correct me above and do it the right way. :) So that it's not biased. Also, read this nice article, that imho nicely compares git and mercurial: http://rg03.wordpress.com/2009/04/07/mercurial-vs-git/ Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
> * I use Python for a bunch of other stuff Matlab is not suitable for -- > This is my argument about usability and tool support. A few years back, > CVS was a standard, now SVN is. I like that I can use the same tool to > contribute to a whole bunch of OS projects, and I use it to manage all > my projects as work. It seems many in the OS world are moving to a DVCS > -- but there are three major ones in play: git, Hg and bzr -- I don't > know enough about any of them to say what I prefer, but I really don't > want to have to learn all three! And if I do, it would be nice if there > were semi-similar interfaces for them all: tortoise, for instance. Unfortunately, one has to learn all 3 (I need to use bzr for ipython, hg for Sage, maybe soon some other projects and git for everything else). But it's not difficult. I think it's like learning bash and then couple commands from tcsh and zsh just to be able to survive. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Fri, Apr 10, 2009 at 1:07 AM, David Cournapeau wrote: > Eric Firing wrote: >> >> This is simply wrong. Mercurial uses hard links for cloning a repo that >> is on the same disk, so it is faster and much more space-efficient than >> copying the files. > > Yes, but maybe Ondrej talks about an older hg version ? Hg could not > handle NTFS hardlink for some time, but it does now if you have pywin32. > > And still, switching between branches is faster in git than in hg, for > some technical reasons I can't really explain (but can be found on the > ML archives). But as I said previously, speed is not really an argument > for me. If hg is fast enough for python, it is obviously fast enough for > numpy and scipy. As long as it does not takes minutes to merge/review > the 5 lines difference between two branches as is the case in svn right > now, I am happier :) > >> But if you do want named branches in a given repo, >> you can have that also with hg. Granted, it has not always been part of >> hg, but it is now. Same with rebasing and transplanting. >> > > As I understand, and correct me if I am wrong, the problems with named > branches are: > - you can't remove them later, it is in the repository 'forever' > - it is not easy to make them publicly available Plus with git, you can fetch the remote repository with all the branches and browse them locally in your remote branches, when you are offline. And merge them with your own branches. In mercurial, it seems the only way to see what changes are there and which branch and which commits I want to merge is to use "hg in", but that requires the internet connection, so it's basically like svn. I don't know if mercurial has improved in this lately, but at least for me, that's a major problem with mercurial. But since python now moved to mercurial too, maybe they will help fix this. :) It seems to me like the python distutils vs cmake discussion. I also prefer the "unpythonic" cmake. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Fri, Apr 10, 2009 at 12:43 AM, Eric Firing wrote: > Ondrej Certik wrote: >> On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau >> wrote: >>> Ondrej Certik wrote: >>>> It is maybe easier to learn how to work with different clones, but >>>> once you start working with lots of patches and you need to reclone >>>> all the time, then it's the wrong approach to work, as it takes lots >>>> of time to copy the whole repository on the disk. > > This is simply wrong. Mercurial uses hard links for cloning a repo that On my laptop, recloning the whole repository (with hardlinks) takes considerably longer than creating a new branch in the same directory, that's a pure fact. > is on the same disk, so it is faster and much more space-efficient than > copying the files. But if you do want named branches in a given repo, > you can have that also with hg. Granted, it has not always been part of > hg, but it is now. Same with rebasing and transplanting. As far as I know mercurial doesn't have interactive rebase. Besides, the default package in Ubuntu Jaunty doesn't even have the (noninteractive) rebase enabled. So I think it will still take quite some (lot?) of time until mercurial has all those tools available by default. > > Now, I know that your (speaking to Ondrej) project switched from hg to > git, and you have provided some useful insight as to why. I also > understand that there are substantive differences between the two, with > advantages and disadvantages. But I don't think it follows that numpy > (or matplotlib, eventually, I hope) necessarily should move to git I never said numpy should move to git because I like git. > if/when a move to a DVCS is decided upon. It is possible that hg might > be a better fit--a better compromise--for present *and* *potential* > *future* contributors to numpy, scipy, and matplotlib. Yes, it may be possible. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Thu, Apr 9, 2009 at 10:45 PM, David Cournapeau wrote: > Ondrej Certik wrote: >> >> It is maybe easier to learn how to work with different clones, but >> once you start working with lots of patches and you need to reclone >> all the time, then it's the wrong approach to work, as it takes lots >> of time to copy the whole repository on the disk. > > Yes, *I* know how to use git, and I agree with you, I vastly prefer git > branch handling to bzr branch handling. *I* find working with GUI for > VCS a real PITA. But I am not the only numpy developer, that's why the > feedback from people like Josef with a totally different workflow than > me is valuable - much more than people like us who are unix geeks :) Yes, definitely. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Thu, Apr 9, 2009 at 10:25 PM, David Cournapeau wrote: > josef.p...@gmail.com wrote: >> So, for my >> style, working with different clones instead of branches seems easier. >> > > Yes, it is. There is no denying that git makes it more difficult for > this workflow, and that git is more difficult at first for easy tasks. I > am interested in making it as easy as possible, and if it is indeed too > difficult, then maybe git is out. It is maybe easier to learn how to work with different clones, but once you start working with lots of patches and you need to reclone all the time, then it's the wrong approach to work, as it takes lots of time to copy the whole repository on the disk. I worked like that with mercurial and I hated it, especially once I started to have ~15 branches. Git branches are cheap and fast. Besides, imho, you can work with different clones with git too, if you want. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Thu, Apr 9, 2009 at 11:16 AM, Andrew Straw wrote: > Matthieu Brucher wrote: >>> One thing about git-svn is that this is not really needed if you just >>> use git and I installed git from source on many linuxes and clusters >>> and it just works, as it is just pure C. I usually just use git-svn on >>> my laptop/workstation, where I install the Debian/Ubuntu packages, and >>> I create the git repository, upload to github.com or somewhere else >>> and just work with the git repository. >>> >>> But I agree that if it installs git-svn and it doesn't just work, it's >>> a big problem. >> >> I was inquiring the use of git with the use of one of our internal svn >> repositories, just to have a feeling about it :( > > My opinion is that attempting to use git-svn to get a feeling of git is > not a good idea. There's too much slowness of svn involved, too much > pain of trying to learn git while also trying to learn git-svn (which > itself has corner cases and such that pure git doesn't) and there's no > bidirectional 1:1 mapping between branches (that I've found), > eliminating a huge part of the joy of git -- cheap branches. > > Better to start developing on a pure git project to get a feel. You > can't go wrong with sympy, for example. :) Oh, definitely. :) Here is a list of easy to fix issues: http://code.google.com/p/sympy/issues/list?q=label:EasyToFix if you want to learn it on some easy, but real world example. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Thu, Apr 9, 2009 at 7:15 AM, Matthieu Brucher wrote: > 2009/4/9 David Cournapeau : >> On Thu, Apr 9, 2009 at 10:38 PM, Matthieu Brucher >> wrote: >>> 2009/4/9 David Cournapeau : >>>> On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: >>>> >>>>> Yes. >>>> >>>> Do you have any windows developers (I am sorry, I am not familiar at >>>> all with sympy)? >>>> >>>> My main concern with git are: >>>> - you may need the command line >>>> - the index can be confusing (you can avoid learning it at first, but >>>> still some errors won't make sense before you get it). >>>> - git is not discoverable (you need to read some documentation) >>> >>> I tried to install git on my computer, as git-svn seemed so nice. I >>> tried git-svn and it told me that some .pm files are missing. Why did >>> it install git-svn if some files are missing? And why isn't it >>> possible to get some information on how to install those missing files >>> on the Internet. >> >> For which system did you install it ? I don't think git-svn work well >> or at all on windows. >> >> Otherwise, git has binary installers for every platform which matters. >> Installing from sources is a bit harder than bzr/hg, but certainly >> easier than svn, if you need to, > > Installed from source on a RH4. It complains about missing SVN/Core.pm. One thing about git-svn is that this is not really needed if you just use git and I installed git from source on many linuxes and clusters and it just works, as it is just pure C. I usually just use git-svn on my laptop/workstation, where I install the Debian/Ubuntu packages, and I create the git repository, upload to github.com or somewhere else and just work with the git repository. But I agree that if it installs git-svn and it doesn't just work, it's a big problem. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Thu, Apr 9, 2009 at 3:04 AM, David Cournapeau wrote: > Ondrej Certik wrote: >> >> Yes, but in fact the staging area (if this is what you mean) is in >> every VCS, only it's hidden, except git, where it is made explicit. >> > > I am not sure the staging area concept is there in other vcs, because in > git it is intrinsically linked to idea that git tracks content. It is If you do "hg add", you add to the staging area. If you do "hg commit", you commit it, together with automatically adding all changes to the staging area as well (and committing). I think it's the same with svn. > powerful, and I really miss it in other DVCS, but it takes time to get > used to - time that other people may not be willing to spend. I tried to > write some basic instructions which totally bypass the index concept, > but it can still bite you even in those simple cases: > > http://projects.scipy.org/numpy/wiki/GitWorkflow#Commonscenario > >> Which tool did you learn? Git? >> > > git. Both Stefan and me really like git, but assuming we make the > transition, we worry a bit about the transition cost for people who are > happy with svn. I think it won't. At least it didn't in the sympy case. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Wed, Apr 8, 2009 at 11:04 PM, David Cournapeau wrote: > On Wed, Apr 8, 2009 at 3:34 AM, Ondrej Certik wrote: > >> Yes. > > Do you have any windows developers (I am sorry, I am not familiar at > all with sympy)? Not much. > > My main concern with git are: > - you may need the command line Oh, definitely. What else? :) > - the index can be confusing (you can avoid learning it at first, but > still some errors won't make sense before you get it). Yes, but in fact the staging area (if this is what you mean) is in every VCS, only it's hidden, except git, where it is made explicit. > - git is not discoverable (you need to read some documentation) Yes, but "git help command" is very detailed most of the time. > > I tend to think those problems are not that big, and the concern > raised on python ML way overblown - but I learned the tool, and I > don't think anyone wants to alienate any numpy/scipy developer. Which tool did you learn? Git? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
2009/4/7 Stéfan van der Walt : > 2009/4/6 Ondrej Certik : >>> FWIW, I tend to agree that Hg is less disruptive than git when coming >>> from svn, at least for the simple tasks (I don't know hg enough to have >>> a really informed opinion for more advanced workflows). >> >> Yes, git rocks. > > Did Sympy move from Hg to Git? Yes. > If so, could you give us an overview > of the pros and cons you experienced? Here are some my (subjective) observations: 1) I think all developers were able to learn git pretty easily. See also: http://wiki.sympy.org/wiki/Git_hg_rosetta_stone 2) git is faster 3) it boosted my productivity a lot, thanks to branches. Previously I was copying the whole directories with sympy, everytime someone sended a patch, just to try it. I know that hg has branches too, but imho working with them is a pain, as you can't rebase patches easily, no interactive rebase etc and also branches were in git from the beginning, so it's very polished. Hg is catching up I guess, but some things are still missing. 4) no need to learn mercurial queues, just learn git and git rebase -i, and that's all I ever need (and it's actually more powerful than MQ). So git is simpler. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
On Mon, Mar 30, 2009 at 10:59 PM, David Cournapeau wrote: > Eric Firing wrote: >> >> I agree. The PEP does not show overwhelming superiority (or, arguably, >> even mild superiority) of any alternative > > I think this PEP was poorly written. You can't see any of the > advantage/differences of the different systems. Some people even said > they don't see the differences with svn. I think the reason partly is > that the PEP focused on existing python workflows, but the whole point, > at least for me, is to change the general workflow (for reviews, code > contributions, etc...). Stephen J. Turnbull sums it up nicely: > > http://mail.python.org/pipermail/python-dev/2009-March/087968.html > > FWIW, I tend to agree that Hg is less disruptive than git when coming > from svn, at least for the simple tasks (I don't know hg enough to have > a really informed opinion for more advanced workflows). Yes, git rocks. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] denormal test: what is it for ?
On Tue, Feb 17, 2009 at 8:08 AM, David Cournapeau wrote: > Hi, > > I would like to continue cleaning the setup.py from numpy/core, and > there is one test that I can't make sense of: the denormal thing > (testcode_mathlib function). The svn log has no information on it (its > presence goes back to the first revision of the file). What is it > useful for ? Denormal support ? Whether exp works ? I guess noone knows. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem looping over numpy array in C
On Fri, Feb 20, 2009 at 12:19 PM, Sumant S.R. Oemrawsingh
wrote:
> Hi guys,
>
> I have a problem with looping over numpy arrays in C. I modify the array
> in-place (and return None), but after modification, the array doesn't seem
> to play nice any more.
>
> Below, I have the C code for a function do_something (a stripped version
> of my original function), which has as arguments a 1D numpy array (float
> or complex) and either an int or a sequence of ints.
>
> In python, I do the following using only the integer:
>
a = array([0., 1., 2., 3., 2., 1., 0.])
do_something(a,3)
savetxt('bla',a)
>
> Which works fine. However, when I do the same, but with a list of any
> length larger than 0:
>
a = array([0., 1., 2., 3., 2., 1., 0.])
do_something(a,[3])
savetxt('bla',a)
> Traceback (most recent call last):
> File "", line 1, in
> File "/usr/lib/python2.5/site-packages/numpy/core/numeric.py", line 767,
> in savetxt
> X = asarray(X)
> File "/usr/lib/python2.5/site-packages/numpy/core/numeric.py", line 132,
> in asarray
> return array(a, dtype, copy=False, order=order)
> TypeError: an integer is required
savetxt('bla',a)
>
> For some reason, the first time it doesn't work; the asarray fails but
> then succeeds on a second try. The resulting file is identical to when I
> would have looped over the list in python, and called the do_something
> function with each integer separately. For instance:
>
> for i in [3,1,0]:
> do_something(a, i)
>
> works fine as well. So apparently looping in C temporarily leaves the
> array in a weird state but manages to automatically be restored after one
> exception. I've checked that type(a), a.dtype, a.shape and a.ndim remain
> the same before and after calling do_something with a sequence or with an
> integer as second argument. That doesn't seem to be the problem.
>
> The reason that I want do the loop in C is that I need some
> precalculations, before being able to do the actual loop. But they might
> not be time-consuming enough to warrant writing it in C, so I could just
> do the loop in python and not have this problem any more. However, if the
> loop in C manages to somehow (temporarily) corrupt the array, how can I be
> sure that the single-shot call doesn't succeed by accident?
>
> If anyone can help, suggest something to try or spot my mistake, I'd
> appreciate it.
I don't have time to go through your code, but I suggest you try
Cython, that plays very nice with numpy.
Ondrej
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy 1.2.1 uploaded to unstable
Hi, I finally found time to upload the numpy 1.2.1 to Debian unstable (currently it's in incoming: http://incoming.debian.org/). The package is lintian clean, but there is one test that failed for me in chroot. I'll wait until it gets to mirrors and then try it on my laptop and report a bug (I uploaded from a ubuntu machine, but of course I compiled it in pbuilder with sid and tested in chroot). Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] parallel compilation of numpy
Hi, I have a shiny new computer with 8 cores and numpy still takes forever to compile --- is there a way to compile it in parallel (make -j9)? Do distutils allow that? If not, let's move to some build system that allows that? Just wanted to check if there is some reason for that, apart from patches welcome. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Mon, Feb 9, 2009 at 2:35 PM, Brian Granger wrote: >> CMake does handle this automatically. >> E.g. if include directories are changed (which you do by editing a >> CMakeLists.txt or the cmake cache), all files which are affected by the are >> rebuilt. If some library changes, everything linking to this library is >> linked again. >> If any of the files the build depends on (e.g. a CMakeLists.txt, or an >> included .cmake file, or the cmake cache) is changed, cmake is automatically >> rerun, and the makefiles/project files are regenerated. >> >> I don't know what happens if you are using both C and Fortran in one project >> and change one of them. I think this is actually not possible, you can (or >> should) not change the compiler of an existing build tree. Basically >> everything which uses this compiler is invalid then, the object files, the >> results of tests etc. I'm not sure handling this separately for different >> languages within one project is supported by cmake. > > Is all of this handled just by calling make (after the initial cmake > call), or do you have to first recall cmake and then make? I just tried that and it is handled automatically by calling "make". It's really cool! Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 7:56 PM, David Cournapeau wrote: > Ondrej Certik wrote: >>> That's exactly what I don't like about cmake - it means you can't >>> produce accurate builds (you need to rerun cmake everytime you change >>> the configuration or dependencies, whereas this is automatic with >>> scons/waf). It also have (used to have) very poor documentation unless >>> you buy the book - but it looks like this is changing. >>> >> >> You can always rerun cmake if you want (and make it automatic). Imho >> that's not a problem. But maybe it is done in a better way with scons. >> > > I think it is a problem - it means you have to update it explicitly when > the configuration changes. In scons, the signature concept is quite > powerful: not only are file dependencies handled, but also command > lines, etc... For example, in numscons, if you change the blas/lapack > from atlas to say MKL, only linking and eventually configuration changes > are rebuilt. If you change the fortran compiler but not the C compiler, > only fortran code is rebuilt. All of this is 100 % automatic. I got an email from Alexander Neundorf from kde, and with his permission, I am forwarding it here, because he is not subscribed. As he writes below, cmake is in fact cabable of the above. -- Forwarded message -- From: Alexander Neundorf Date: Mon, Feb 9, 2009 at 1:28 PM Subject: cmake information To: Ondrej Certik Hi Ondrej, I hope you are the Ondrej from this thread: http://groups.google.com/group/Numpy-discussion/browse_thread/thread/c12f96b9ac367f57 If not, please let me know and I'll try to find the right email address. I saw this in the thread: --8<-8<8< Ondrej Certik wrote: >> That's exactly what I don't like about cmake - it means you can't >> produce accurate builds (you need to rerun cmake everytime you change >> the configuration or dependencies, whereas this is automatic with >> scons/waf). It also have (used to have) very poor documentation unless >> you buy the book - but it looks like this is changing. > You can always rerun cmake if you want (and make it automatic). Imho > that's not a problem. But maybe it is done in a better way with scons. I think it is a problem - it means you have to update it explicitly when the configuration changes. In scons, the signature concept is quite powerful: not only are file dependencies handled, but also command lines, etc... For example, in numscons, if you change the blas/lapack from atlas to say MKL, only linking and eventually configuration changes are rebuilt. If you change the fortran compiler but not the C compiler, only fortran code is rebuilt. All of this is 100 % automatic. --8<-8<8< CMake does handle this automatically. E.g. if include directories are changed (which you do by editing a CMakeLists.txt or the cmake cache), all files which are affected by the are rebuilt. If some library changes, everything linking to this library is linked again. If any of the files the build depends on (e.g. a CMakeLists.txt, or an included .cmake file, or the cmake cache) is changed, cmake is automatically rerun, and the makefiles/project files are regenerated. I don't know what happens if you are using both C and Fortran in one project and change one of them. I think this is actually not possible, you can (or should) not change the compiler of an existing build tree. Basically everything which uses this compiler is invalid then, the object files, the results of tests etc. I'm not sure handling this separately for different languages within one project is supported by cmake. Alex ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: HDF5 for Python 1.1
On Mon, Feb 9, 2009 at 12:15 PM, Andrew Collette wrote: > = > Announcing HDF5 for Python (h5py) 1.1 > = > > What is h5py? > - > > HDF5 for Python (h5py) is a general-purpose Python interface to the > Hierarchical Data Format library, version 5. HDF5 is a versatile, > mature scientific software library designed for the fast, flexible > storage of enormous amounts of data. Interesting project. My first question was how it is related to pytables. The answer is here: http://code.google.com/p/h5py/wiki/FAQ#What%27s_the_difference_between_h5py_and_Pytables? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
> That's exactly what I don't like about cmake - it means you can't > produce accurate builds (you need to rerun cmake everytime you change > the configuration or dependencies, whereas this is automatic with > scons/waf). It also have (used to have) very poor documentation unless > you buy the book - but it looks like this is changing. You can always rerun cmake if you want (and make it automatic). Imho that's not a problem. But maybe it is done in a better way with scons. > >> What I don't like on cmake is that it uses it's own language, instead >> of python, on the other hand, so far everything seems to just work. >> Contrary to numscons, where it looks almost like a new python program >> just to build numpy. > > Again, numscons is just a library on top of scons to support things we > need in numpy, it is not really a new program - it is a separate > package to avoid adding experimental code to numpy itself. Numscons is > ~ 3000 LOC, of which 1000 is for the core, 1000 for > blas/lapack/fortran support, and 1000 for tools which are not properly > supported in scons (recent MSVC, python extensions). > > I think you would have almost as much work with cmake if not more - > when I started numscons, cmake did not have fortran support (it now > has, although I don't know how far - it does not seem to handle mixed > fortran/C, for example). > > If you don't mind fast changing software, you should look at waf: it > is extremely fast, based on python. It also handles a lot of > distribution issues already (tarball generation, compiler selection, > etc...) which scons does not. Yes, waf is pretty cool, even though the last time I looked at it, it wasn't able to compile my project, which was larger than just couple files. But it seems it's development is progressing fast. As to the other things, one nice thing about cmake is that it is production ready right now, it is well tested (kde4) and it is in distributions (e.g. Debian). Everything else needs nontrivial adjustments, or is not as well tested (yet). Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 12:26 PM, Brian Granger wrote: >> Yes, I am investigating cmake, it's pretty cool. I wrote some macros >> for cython etc. What I like about cmake is that it is cross platform >> and it just produces makefiles on linux, or visual studio files (or >> whatever) on windows. When I get more experience with it, I'll post >> here. > > Yes, while I haven't played with cmake much, it does look very nice. > >> What I don't like on cmake is that it uses it's own language, instead >> of python, on the other hand, so far everything seems to just work. > > I too don't like the idea of cmake having its own language. While I > love to learn new languages, I like to have good reasons and I'm not > sure that building software is a good reason. > > I have been playing with Scons and I really do like the fact that it > is Python. While I haven't tried it yet, I am pretty sure that we > could use IPython to interactively debug a Scons build. That would be > really nice! > >> Contrary to numscons, where it looks almost like a new python program >> just to build numpy. > > But hold on, it is not fair to compare cmake with numscons (this is an > apples and oranges thing). You should compare cmake with *Scons* > itself. The problem with comparing cmake with numscons is that cmake > can't do what numscons does - i.e., plug into distutils. There is a > lot of extra things that distutils does other than build extensions > and cmake won't do these things out of the box. Obviously, someone > could get cmake to do these things, but then you are developing a > complete distutils replacement. And I think that any distutils > replacement should done in Python. Yes, I agree with what you said. I still want to try cmake though, if nothing, at least to know where we want to go. Yes, I should compare cmake with scons. We can either develop python front end to cmake, or just help improve scons/numscons etc. I really like that cmake generates native makefiles and windows visual studio files etc. In any case, whatever we choose to use, I want to have experience with both scons and cmake. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 3:10 AM, David Cournapeau wrote: > On Sun, Feb 8, 2009 at 3:21 AM, Ondrej Certik wrote: >> Hi David, >> >>> Sorry for the confusion: numscons is NOT the preferred build system. >>> The current numpy.distutils extensions, as shipped by numpy, is the >>> preferred one. Numscons is more an experiment, if you want. >> >> Ah, I see, thanks for the clarification. >> >>>> So is it supposed to be in Debian? >>> >>> No, I don't think it should be. It is not yet stabilized code wise, so >>> it does not make much sense to package it. >> >> Ok. >> >>> >>>> Is numscons supposed to be >>>> a build system for other projects as well? Why not to just send the >>>> needed patches to scons and just use scons? >>> >>> Because you cannot just use scons. Numscons is a library build on top >>> of scons, for the needs of numpy. There also needs to be some hook >>> from numpy.distutils to use scons (numscons adds a new distutils >>> command, which is used instead of build to build any compiled >>> code-based extensions). Most of the changes needed for scons have been >>> integrated upstream, though, except one or two things. >> >> I see. I think it's a bit confusing that one needs to build a new >> build system just to build numpy, e.g. that both distutils and scons >> are not good enough. > > I don't find it that surprising - numpy and scipy require some > relatively advanced features (mixed language and cross-platform with > support for many toolchains). Within the open source tools, I know > only two which can handle those requirements: scons and cmake. For > example, it would almost impossible to build numpy/scipy with > autoconf. Yes, I am investigating cmake, it's pretty cool. I wrote some macros for cython etc. What I like about cmake is that it is cross platform and it just produces makefiles on linux, or visual studio files (or whatever) on windows. When I get more experience with it, I'll post here. What I don't like on cmake is that it uses it's own language, instead of python, on the other hand, so far everything seems to just work. Contrary to numscons, where it looks almost like a new python program just to build numpy. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
Hi David, > Sorry for the confusion: numscons is NOT the preferred build system. > The current numpy.distutils extensions, as shipped by numpy, is the > preferred one. Numscons is more an experiment, if you want. Ah, I see, thanks for the clarification. >> So is it supposed to be in Debian? > > No, I don't think it should be. It is not yet stabilized code wise, so > it does not make much sense to package it. Ok. > >> Is numscons supposed to be >> a build system for other projects as well? Why not to just send the >> needed patches to scons and just use scons? > > Because you cannot just use scons. Numscons is a library build on top > of scons, for the needs of numpy. There also needs to be some hook > from numpy.distutils to use scons (numscons adds a new distutils > command, which is used instead of build to build any compiled > code-based extensions). Most of the changes needed for scons have been > integrated upstream, though, except one or two things. I see. I think it's a bit confusing that one needs to build a new build system just to build numpy, e.g. that both distutils and scons are not good enough. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] preferred numpy build system
Hi,
I have couple beginners questions about numscons. What is the
preferred build system for numpy now, is it numscons? The README
doesn't mention numscons, so I am a bit confused what the future plan
is.
Also by doing:
$ python setupscons.py install
Running from numpy source directory.
Traceback (most recent call last):
File "setupscons.py", line 56, in
raise DistutilsError('\n'.join(msg))
distutils.errors.DistutilsError: You cannot build numpy with scons
without the numscons package
(Failure was: No module named numscons)
so the numscons package needs to be installed -- but it is not in
Debian. So is it supposed to be in Debian? Is numscons supposed to be
a build system for other projects as well? Why not to just send the
needed patches to scons and just use scons?
Thanks,
Ondrej
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] porting NumPy to Python 3
Hi James, On Thu, Jan 29, 2009 at 2:11 AM, James Watson wrote: > Hi, > > I am interested in contributing to the port of NumPy to Python 3. Who > I should coordinate effort with? > > I have started at the Python end of the problem (as opposed to > http://www.scipy.org/Python3k), e.g. I have several patches to get > 2to3 to work on NumPy's Python source code. I am sorry that noone has replied to your email. Could you please upload your patches somewhere? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] whohas command
Hi, I just discovered a new whohas command in Debian, that can show a package in virtually all important distributions (Arch, Debian, Fedora, Gentoo, openSUSE, Slackware (and linuxpackages.net), Source Mage, Ubuntu, FreeBSD, NetBSD, OpenBSD, Fink and MacPorts distributions). So for example SymPy is in a lot of them already: $ whohas sympy Archpython-sympy-git 20090121-1 unsupported http://aur.archlinux.org/packages.php?ID=23341 MacPortspy-sympy 0.5.15 http://trac.macports.org/browser/trunk/dports/python/py-sympy/Portfile MacPortspy25-sympy0.6.3 http://trac.macports.org/browser/trunk/dports/python/py25-sympy/Portfile FreeBSD py25-sympy0.6.3 math http://www.freebsd.org/cgi/pds.cgi?ports/math/py25-sympy NetBSD py-sympy 0.5.15 mathhttp://pkgsrc.se/math/py-sympy Ubuntu python-sympy 0.4.2-1 universehttp://packages.ubuntu.com/gutsy/python-sympy Ubuntu python-sympy 0.5.8-1 universehttp://packages.ubuntu.com/hardy/python-sympy Ubuntu python-sympy 0.5.15-1~hardy1 universe http://packages.ubuntu.com/hardy-backports/python-sympy Ubuntu python-sympy 0.5.15-1 universehttp://packages.ubuntu.com/intrepid/python-sympy Ubuntu python-sympy 0.6.3-1 universehttp://packages.ubuntu.com/jaunty/python-sympy Debian python-sympy 0.6.1-1 testing http://packages.debian.org/lenny/python-sympy Finksympy-py240.5.6-1 http://pdb.finkproject.org/pdb/package.php/sympy-py24 Finksympy-py250.5.6-1 http://pdb.finkproject.org/pdb/package.php/sympy-py25 Gentoo sympy 0.6.3 http://gentoo-portage.com/dev-python/sympy Gentoo sympy 0.6.2 http://gentoo-portage.com/dev-python/sympy openSUSEpython-sympy 0.6.3 science http://packages.opensuse-community.org/packageinfo.jsp?checksum=a2258d3deaee5f411456f22ea1ab6bb7ec95e12c&distro=openSUSE_111 It's also interesting to click on the links to get to the pages of sympy for each distribution. And numpy: $ whohas numpy Archpython-numexpr1.1.1-1 unsupported http://aur.archlinux.org/packages.php?ID=23380 Archpython-numpy 1.2.1-3 community http://aur.archlinux.org/packages.php?ID=8497 Archscikits-audiolab-svn 1491-1 unsupported http://aur.archlinux.org/packages.php?ID=21018 FreeBSD py25-numpy1.2.1,1 math http://www.freebsd.org/cgi/pds.cgi?ports/math/py25-numpy FreeBSD py25-symeig 1.4_1 math http://www.freebsd.org/cgi/pds.cgi?ports/math/py25-symeig NetBSD py-numpy 1.1.0 wip http://pkgsrc.se/wip/py-numpy NetBSD py-numpy 1.1.0 mathhttp://pkgsrc.se/math/py-numpy Ubuntu python-numpy 1:1.0.1-8ubuntu1 universehttp://packages.ubuntu.com/feisty/python-numpy Ubuntu python-numpy 1:1.0.3-1ubuntu2 universehttp://packages.ubuntu.com/gutsy/python-numpy Ubuntu python-numpy 1:1.0.4-6ubuntu3 universehttp://packages.ubuntu.com/hardy/python-numpy Ubuntu python-numpy 1:1.1.1-1 http://packages.ubuntu.com/intrepid/python-numpy Ubuntu python-numpy 1:1.1.1-2ubuntu1 http://packages.ubuntu.com/jaunty/python-numpy Ubuntu python-numpy-dbg 1:1.0.1-8ubuntu1 universehttp://packages.ubuntu.com/feisty/python-numpy-dbg Ubuntu python-numpy-dbg 1:1.0.3-1ubuntu2 universehttp://packages.ubuntu.com/gutsy/python-numpy-dbg Ubuntu python-numpy-dbg 1:1.0.4-6ubuntu3 universehttp://packages.ubuntu.com/hardy/python-numpy-dbg Ubuntu python-numpy-dbg 1:1.1.1-1 http://packages.ubuntu.com/intrepid/python-numpy-dbg Ubuntu python-numpy-dbg 1:1.1.1-2ubuntu1 http://packages.ubuntu.com/jaunty/python-numpy-dbg Ubuntu python-numpy-dev 1:1.0.1-8ubuntu1 universehttp://packages.ubuntu.com/feisty/python-numpy-dev Ubuntu python-numpy-dev 1:1.0.3-1ubuntu2 universehttp://packages.ubuntu.com/gutsy/python-numpy-dev Ubuntu python-numpy-dev http://packages.ubuntu.com/hardy/python-numpy-dev Ubuntu python-numpy-doc 1:1.0.1-8ubuntu1 universehttp://packages.ubuntu.com/feisty/python-numpy-doc Ubuntu python-numpy-doc 1:1.0.3-1ubuntu2 universehttp://packages.ubuntu.com/gutsy/python-numpy-doc Ubuntu python-numpy-doc 1:1.0.4-6ubuntu3 universehttp://packages.ubuntu.com/hardy/python-numpy-doc Ubuntu python-numpy-doc 1:1.1.1-1 http://pac
Re: [Numpy-discussion] missing doc dir in the official tarball
On Sun, Dec 21, 2008 at 1:49 PM, Pauli Virtanen wrote: > Sun, 21 Dec 2008 13:05:57 +0100, Ondrej Certik wrote: > >> On Sat, Dec 20, 2008 at 3:02 PM, Pauli Virtanen wrote: >>> Sat, 20 Dec 2008 20:15:43 +0900, David Cournapeau wrote: >>>> On Sat, Dec 20, 2008 at 7:43 PM, Ondrej Certik >>>> wrote: >>>>> Just to make it clear -- I think the docs should not be generated in >>>>> the tarball -- only the sources should be there. >>>> >>>> I agree this makes more sense for you, as a packager, but I am not >>>> sure it makes much sense to put the doc sources in the tarball for >>>> users (Building numpy should only require python + a C compiler; >>>> building the doc is more difficult -you need at least sphinx and all >>>> its dependencies). >>>> >>>> For audiolab, I put the generated doc, thinking if people want to mess >>>> with the doc, they are knowledgeable enough to deal with svn - but I >>>> did not think about the packagers :) I am not sure what's the best >>>> solution: maybe put both in the (released) source tarball ? >>> >>> I'd say that we put the source for the documentation to the >>> documentation tarball, and distribute the built HTML+whatever >>> documentation in a separate package. >> >> Why not to just include the *sources* together with numpy, and possibly >> include html+whatever in a separate documentation package? > > That's what I tried to say, but mistyped "source" as "documentation". Ok, so we all seem to agree that having (at least) the source of docs together with the main numpy tarball is a good thing. I'll try to have a look at this. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] missing doc dir in the official tarball
On Sat, Dec 20, 2008 at 3:02 PM, Pauli Virtanen wrote: > Sat, 20 Dec 2008 20:15:43 +0900, David Cournapeau wrote: >> On Sat, Dec 20, 2008 at 7:43 PM, Ondrej Certik wrote: >>> Just to make it clear -- I think the docs should not be generated in >>> the tarball -- only the sources should be there. >> >> I agree this makes more sense for you, as a packager, but I am not sure >> it makes much sense to put the doc sources in the tarball for users >> (Building numpy should only require python + a C compiler; building the >> doc is more difficult -you need at least sphinx and all its >> dependencies). >> >> For audiolab, I put the generated doc, thinking if people want to mess >> with the doc, they are knowledgeable enough to deal with svn - but I did >> not think about the packagers :) I am not sure what's the best solution: >> maybe put both in the (released) source tarball ? > > I'd say that we put the source for the documentation to the documentation > tarball, and distribute the built HTML+whatever documentation in a > separate package. Why not to just include the *sources* together with numpy, and possibly include html+whatever in a separate documentation package? That way everybody is happy. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] missing doc dir in the official tarball
On Sat, Dec 20, 2008 at 3:35 AM, David Cournapeau wrote: > On Sat, Dec 20, 2008 at 7:43 AM, Stéfan van der Walt wrote: >> 2008/12/20 Ondrej Certik : >>> So we thought with Stefan that maybe a simpler solution is just to fix >>> the ./setup sdist (or how you create the tarball in numpy) to include >>> documentation and be done with it. >> >> I think releases should either include the Sphinx documentation or, >> alternatively, we should provide a separate tar-ball for the docs >> along with every release. > > How difficult would it be to generate the doc ? I have not followed in > detail what happened recently on that front. Is sphinx + sphinx ext > the only necessary additional tools ? Just to make it clear -- I think the docs should not be generated in the tarball -- only the sources should be there. The html (and/or pdf) docs will be generated at the package build. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Missing numpy.i
On Tue, Nov 18, 2008 at 11:42 AM, Nicolas ROUX wrote: > Hi, > > About the missing doc directory in the windows install in latest numpy > release, could you please add it ? > (please see below the previous thread) Well, this is a serious problem, so it should definitely be fixed, see here: http://projects.scipy.org/pipermail/numpy-discussion/2008-December/039309.html Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] missing doc dir in the official tarball
Hi, while packaging the new version of numpy, I realized that it is missing a documentation. I just checked with Stefan on Jabber and he thinks it should be rather a trivial fix. Do you Jarrod think you could please release a new tarball with the doc directory? The problem is that debian (and I guess other distros as well) has one source package (e.g. numpy tarball + debian files) and it creates python-numpy, python-numpy-dbg and python-numpy-doc binary packages from it. There should definitely be a doc package. So if the tarball is missing documentation, we need to repackage it. Since the doc is only in svn (right?), we would have to write some scripts to first svn checkout the doc, unpack the official tarball, include the doc, pack it and that would be our tarball. So we thought with Stefan that maybe a simpler solution is just to fix the ./setup sdist (or how you create the tarball in numpy) to include documentation and be done with it. What do you think? If you are busy, I can look at it how to fix the numpy tarball creation. Thanks, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] python-numpy: memory leak in exponentiation
Hi, the details including a test script are at: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=505999 Thanks, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Fri, Sep 19, 2008 at 10:37 AM, Robert Cimrman <[EMAIL PROTECTED]> wrote: > Ondrej Certik wrote: >> On Thu, Sep 18, 2008 at 4:12 PM, Ryan May <[EMAIL PROTECTED]> wrote: >>> Ondrej Certik wrote: >>>> On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman <[EMAIL PROTECTED]> wrote: >>>>>> It requires Cython and a C compiler to build. I'm still debating >>>>>> myself about the desired workflow for using it, but for now, it only >>>>>> profiles functions which you have registered with it. I have made the >>>>>> profiler work as a decorator to make this easy. E.g., >>>>> many thanks for this! I have wanted to try out the profiler but failed >>>>> to build it (changeset 6 0de294aa75bf): >>>>> >>>>> $ python setup.py install --root=/home/share/software/ >>>>> running install >>>>> running build >>>>> running build_py >>>>> creating build >>>>> creating build/lib.linux-i686-2.4 >>>>> copying line_profiler.py -> build/lib.linux-i686-2.4 >>>>> running build_ext >>>>> cythoning _line_profiler.pyx to _line_profiler.c >>>>> building '_line_profiler' extension >>>>> creating build/temp.linux-i686-2.4 >>>>> i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC >>>>> -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c >>>>> -o build/temp.linux-i686-2.4/_line_profiler.o >>>>> _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a >>>>> function) >>>>> error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 >>>>> >>>>> I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. >>>> I am telling you all the time Robert to use Debian that it just works >>>> and you say, no no, gentoo is the best. :) >>> And what's wrong with that? :) Once you get over the learning curve, >>> Gentoo works just fine. Must be Robert K.'s fault. :) >> >> Well, I think if Robert C. hasn't yet get over the learning curve >> after so many years of hard work, maybe the learning curve is too >> steep. :) > > This is most probably not related to Gentoo at all and certainly not > related to me knowing Gentoo or not :) (and no, learning Gentoo is not > that hard.) Let us know where the problem was. :) I am just using common sense, if something works on Debian and macosx and doesn't work on gentoo, I thought it was safe to say it was gentoo related, but I may well be wrong. :)) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
> So the timing raises a lot. For obvious reasons, that's the overhead > of the profiler. But the problem is that then the timings just don't > fit, e.g. if I sum the total time spent in subfunctions, it doesn't > account for all the time printed on the respective line in the parent > function. > > I don't know if there is any way to fix it, or even worth fixing. So I > guess one should just use the profiling info to determine which line > to fix. > > Do you think it's worthy to implement line profiling for all lines and > then make sure that all timings match? What is your experience. The reason I want this is so that I can determine just by looking at the timing how much time is spent at the line and how much time is spent at the functions that are being called at the line. I think it is doable, what do you think? One would trace how much time was wasted in python_trace_callback and then just substract this time from all parent function's lines timings. Btw Robin, how is Matlab doing it? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Thu, Sep 18, 2008 at 4:12 PM, Ryan May <[EMAIL PROTECTED]> wrote: > Ondrej Certik wrote: >> On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman <[EMAIL PROTECTED]> wrote: >>>> >>>> It requires Cython and a C compiler to build. I'm still debating >>>> myself about the desired workflow for using it, but for now, it only >>>> profiles functions which you have registered with it. I have made the >>>> profiler work as a decorator to make this easy. E.g., >>> many thanks for this! I have wanted to try out the profiler but failed >>> to build it (changeset 6 0de294aa75bf): >>> >>> $ python setup.py install --root=/home/share/software/ >>> running install >>> running build >>> running build_py >>> creating build >>> creating build/lib.linux-i686-2.4 >>> copying line_profiler.py -> build/lib.linux-i686-2.4 >>> running build_ext >>> cythoning _line_profiler.pyx to _line_profiler.c >>> building '_line_profiler' extension >>> creating build/temp.linux-i686-2.4 >>> i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC >>> -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c >>> -o build/temp.linux-i686-2.4/_line_profiler.o >>> _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a >>> function) >>> error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 >>> >>> I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. >> >> I am telling you all the time Robert to use Debian that it just works >> and you say, no no, gentoo is the best. :) > > And what's wrong with that? :) Once you get over the learning curve, > Gentoo works just fine. Must be Robert K.'s fault. :) Well, I think if Robert C. hasn't yet get over the learning curve after so many years of hard work, maybe the learning curve is too steep. :) Anyway, back to work: Robert K., I noticed that if I profile some function, I get results like this for example: 40 307246952 6.6 [x,w] = p_roots(n) 41 307224192 3.4 x = real(x) 42 307234470 4.8 ainf, binf = map(isinf,(a,b)) 43 3072 6565 0.9 if ainf or binf: 44 raise ValueError, "Gaussian quadrature is only available for " \ 45"finite limits." 46 3072 5093 0.7 if not reference: 47 x = (b-a)*(x+1)/2.0 + a 48 3072 594190 83.5 return (b-a)/2.0*sum(w*func(x,*args),0) --- Then if I turn profiling of the func() method, I get this: 40 307246999 4.6 [x,w] = p_roots(n) 41 307224313 2.4 x = real(x) 42 307234327 3.4 ainf, binf = map(isinf,(a,b)) 43 3072 6190 0.6 if ainf or binf: 44 raise ValueError, "Gaussian quadrature is only available for " \ 45"finite limits." 46 3072 4918 0.5 if not reference: 47 x = (b-a)*(x+1)/2.0 + a 48 3072 906876 88.6 return (b-a)/2.0*sum(w*func(x,*args),0) --- So the timing raises a lot. For obvious reasons, that's the overhead of the profiler. But the problem is that then the timings just don't fit, e.g. if I sum the total time spent in subfunctions, it doesn't account for all the time printed on the respective line in the parent function. I don't know if there is any way to fix it, or even worth fixing. So I guess one should just use the profiling info to determine which line to fix. Do you think it's worthy to implement line profiling for all lines and then make sure that all timings match? What is your experience. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Thu, Sep 18, 2008 at 1:01 PM, Robert Cimrman <[EMAIL PROTECTED]> wrote: > Hi Robert, > > Robert Kern wrote: >> On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg <[EMAIL PROTECTED]> wrote: >>> That would make me an extremely happy user, I've been looking for this for >>> years! >>> I can't imagine I'm the only one who profiles some hundred lines of code and >>> ends up with 90% of total time in the dot-function >> >> For the time being, you can grab it here: >> >> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ >> >> It requires Cython and a C compiler to build. I'm still debating >> myself about the desired workflow for using it, but for now, it only >> profiles functions which you have registered with it. I have made the >> profiler work as a decorator to make this easy. E.g., > > many thanks for this! I have wanted to try out the profiler but failed > to build it (changeset 6 0de294aa75bf): > > $ python setup.py install --root=/home/share/software/ > running install > running build > running build_py > creating build > creating build/lib.linux-i686-2.4 > copying line_profiler.py -> build/lib.linux-i686-2.4 > running build_ext > cythoning _line_profiler.pyx to _line_profiler.c > building '_line_profiler' extension > creating build/temp.linux-i686-2.4 > i486-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC > -I/usr/include/python2.4 -c -I/usr/include/python2.4 -c _line_profiler.c > -o build/temp.linux-i686-2.4/_line_profiler.o > _line_profiler.c:1614: error: 'T_LONGLONG' undeclared here (not in a > function) > error: command 'i486-pc-linux-gnu-gcc' failed with exit status 1 > > I have cython-0.9.8.1 and GCC 4.1.2, 32-bit machine. I am telling you all the time Robert to use Debian that it just works and you say, no no, gentoo is the best. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] profiling line by line
On Thu, Sep 18, 2008 at 1:29 AM, Robert Kern <[EMAIL PROTECTED]> wrote:
> On Wed, Sep 17, 2008 at 18:09, Ondrej Certik <[EMAIL PROTECTED]> wrote:
>> On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern <[EMAIL PROTECTED]> wrote:
>>> On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg <[EMAIL PROTECTED]> wrote:
>>>> That would make me an extremely happy user, I've been looking for this for
>>>> years!
>>>> I can't imagine I'm the only one who profiles some hundred lines of code
>>>> and
>>>> ends up with 90% of total time in the dot-function
>>>
>>> For the time being, you can grab it here:
>>>
>>> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/
>>>
>>> It requires Cython and a C compiler to build. I'm still debating
>>> myself about the desired workflow for using it, but for now, it only
>>> profiles functions which you have registered with it. I have made the
>>> profiler work as a decorator to make this easy. E.g.,
>>>
>>> from line_profiler import LineProfiler
>>>
>>> profile = LineProfiler()
>>>
>>> @profile
>>> def my_slow_func():
>>> ...
>>>
>>> profile.dump_stats('my_slow_func.lprof')
>>>
>>> This is kind of inconvenient, so I have a little script that handles
>>> everything except for the @profile itself. It started as a script to
>>> run cProfile nicely, so it actually does that by default.
>>>
>>> I took pystone.py from the Python source and added a couple of
>>> @profile decorators to demonstrate:
>>>
>>>
>>> [line_profiler]$ ./kernprof.py --help
>>> Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg]
>>> ...
>>>
>>> Options:
>>> -h, --helpshow this help message and exit
>>> -l, --line-by-lineUse the line-by-line profiler from the line_profiler
>>>module instead of cProfile. Implies --builtin.
>>> -b, --builtin Put 'profile' in the builtins. Use 'profile.enable()'
>>>and 'profile.disable()' in your code to turn it on
>>> and
>>>off, or '@profile' to decorate a single function, or
>>>'with profile:' to profile a single section of code.
>>> -o OUTFILE, --outfile=OUTFILE
>>>Save stats to
>>> -s SETUP, --setup=SETUP
>>>Code to execute before the code to profile
>>>
>>> [line_profiler]$ ./kernprof.py -l pystone.py
>>> Pystone(1.1) time for 5 passes = 11.34
>>> This machine benchmarks at 4409.17 pystones/second
>>> Wrote profile results to pystone.py.lprof
>>>
>>> [line_profiler]$ ./view_line_prof.py pystone.py.lprof
>>> Timer unit: 1e-06 s
>>>
>>> File: pystone.py
>>> Function: Proc0 at line 79
>>> Total time: 8.46516 s
>> [...]
>>
>> This is what I am getting:
>>
>> $ ./kernprof.py -l pystone.py
>> Wrote profile results to pystone.py.lprof
>> $ ./view_line_prof.py pystone.py.lprof
>> Timer unit: 1e-06 s
>>
>> $
>>
>> So I think you meant:
>>
>> $ ./kernprof.py -l mystone.py
>> 20628
>> Wrote profile results to mystone.py.lprof
>> $ ./view_line_prof.py mystone.py.lprof
>> Timer unit: 1e-06 s
>>
>> File: pystone.py
>> Function: Proc0 at line 79
>> Total time: 13.0803 s
>> [...]
>>
>> Now it works.
>
> No, I meant pystone.py. My script-finding code may have (incorrectly)
> found a different, uninstrumented pystone.py file somewhere else,
> though. Try with "./pystone.py".
>
>> This is an excellent piece of software! Nice job. Just today I needed
>> such a thing!
>>
>>
>> How do you easily install it?
>
> "python setup.py install" should have installed the module. I haven't
> done anything with the scripts, yet.
>
>> I usually do "python setup.py install
>> --home=~/lib" and I have the PYTHONPATH + PATH setup in my .bashrc,
>> but I then need to manually remove the stuff from my ~/lib if I want
>> to uninstall, which sucks. So this time I just did "python setup.py
>> build" and moved the .so file manually to the current dir. But there
>> must be a better way. What is your workflow?
>
> For things I am develop
Re: [Numpy-discussion] profiling line by line
On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern <[EMAIL PROTECTED]> wrote:
> On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg <[EMAIL PROTECTED]> wrote:
>> That would make me an extremely happy user, I've been looking for this for
>> years!
>> I can't imagine I'm the only one who profiles some hundred lines of code and
>> ends up with 90% of total time in the dot-function
>
> For the time being, you can grab it here:
>
> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/
>
> It requires Cython and a C compiler to build. I'm still debating
> myself about the desired workflow for using it, but for now, it only
> profiles functions which you have registered with it. I have made the
> profiler work as a decorator to make this easy. E.g.,
>
> from line_profiler import LineProfiler
>
> profile = LineProfiler()
>
> @profile
> def my_slow_func():
> ...
>
> profile.dump_stats('my_slow_func.lprof')
>
> This is kind of inconvenient, so I have a little script that handles
> everything except for the @profile itself. It started as a script to
> run cProfile nicely, so it actually does that by default.
>
> I took pystone.py from the Python source and added a couple of
> @profile decorators to demonstrate:
>
>
> [line_profiler]$ ./kernprof.py --help
> Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ...
>
> Options:
> -h, --helpshow this help message and exit
> -l, --line-by-lineUse the line-by-line profiler from the line_profiler
>module instead of cProfile. Implies --builtin.
> -b, --builtin Put 'profile' in the builtins. Use 'profile.enable()'
>and 'profile.disable()' in your code to turn it on and
>off, or '@profile' to decorate a single function, or
>'with profile:' to profile a single section of code.
> -o OUTFILE, --outfile=OUTFILE
>Save stats to
> -s SETUP, --setup=SETUP
>Code to execute before the code to profile
>
> [line_profiler]$ ./kernprof.py -l pystone.py
> Pystone(1.1) time for 5 passes = 11.34
> This machine benchmarks at 4409.17 pystones/second
> Wrote profile results to pystone.py.lprof
>
> [line_profiler]$ ./view_line_prof.py pystone.py.lprof
> Timer unit: 1e-06 s
>
> File: pystone.py
> Function: Proc0 at line 79
> Total time: 8.46516 s
[...]
This is what I am getting:
$ ./kernprof.py -l pystone.py
Wrote profile results to pystone.py.lprof
$ ./view_line_prof.py pystone.py.lprof
Timer unit: 1e-06 s
$
So I think you meant:
$ ./kernprof.py -l mystone.py
20628
Wrote profile results to mystone.py.lprof
$ ./view_line_prof.py mystone.py.lprof
Timer unit: 1e-06 s
File: pystone.py
Function: Proc0 at line 79
Total time: 13.0803 s
[...]
Now it works.
This is an excellent piece of software! Nice job. Just today I needed
such a thing!
How do you easily install it? I usually do "python setup.py install
--home=~/lib" and I have the PYTHONPATH + PATH setup in my .bashrc,
but I then need to manually remove the stuff from my ~/lib if I want
to uninstall, which sucks. So this time I just did "python setup.py
build" and moved the .so file manually to the current dir. But there
must be a better way. What is your workflow?
Anyway, so I used it on my code and here is what I got:
File: hermes1d/mesh.py
Function: integrate_function at line 119
Total time: 0.647412 s
Line # Hits Time % Time Line Contents
=
119 @profile
120 def integrate_function(self, f):
121 """
122 Integrate the function
"f" on the element.
123 """
12496 1091 0.2 from numpy import array
12596 461070 71.2 from scipy.integrate
import quadrature
12696 496 0.1 a, b =
self.nodes[0].x, self.nodes[1].x
12796 418 0.1 def func(x):
128 #print x
129 #print
array([f.f(y) for y in x])
130 return
array([f.f(y) for y in x])
13196 183903 28.4 val, err =
quadrature(func, a, b)
132 #print val, a, b
133 #stop
13496 434 0.1 return val
This is interesting, because 70% of time is spent on "from
scipy.integrate import quadrature". So this is not good.
So I moved it out and voila! My code is 3.4x faster. I really didn't
know that importing (scipy.integrate) is *so* slow.
File: hermes1d/mesh.py
Function: integrate_function at line 121
Total time: 0.18
Re: [Numpy-discussion] Updated Numpy reference guide
Hi Pauli! On Sun, Aug 31, 2008 at 8:21 PM, Pauli Virtanen <[EMAIL PROTECTED]> wrote: > > Hi all, > > I finished the first iteration of incorporating material from Travis > Oliphant's "Guide to Numpy" to the Sphinxy reference guide we were > constructing in the Doc marathon. > > Result is here: (the PDF is a bit ugly, though, some content is almost > randomly scattered there) > >http://www.iki.fi/pav/tmp/numpy-refguide/index.xhtml >http://www.iki.fi/pav/tmp/numpy-refguide/NumPy.pdf > > Source is here: (Stéfan, if it looks ok to you, could you pull and check > if it builds for you when you have time?) > >https://code.launchpad.net/~pauli-virtanen/scipy/numpy-refguide > > What I did with the "Guide to Numpy" material was: > > - Collapsed each of the reference Chapters 3, 6, 8, 9 (ndarrays, scalars, > dtypes, ufuncs) with the more introductory material in Chapter 2. > > - As this was supposed to be a reference guide, I tried to compress the > text from Chapter 2 as much as possible, by sticking to definitions and > dropping some more tutorial-oriented parts. This may have reduced > readability at some points... > > - I added some small bits or rewrote parts in the above sections in > places where I thought it would improve the result. > > - I did not include material that I thought was better to be put into > appropriate docstrings in Numpy. > > What to do with class docstrings and obscure __xxx__ attributes was not > so clear a decision, so what I did for these varies. > > - The sections about Ufuncs and array indexing are taken almost verbatim > from the "Guide to Numpy". The ndarray, scalar and dtype sections > somewhat follow the structure of the Guide, but the text is more heavily > edited from the original. > > Some things to do: > > - Descriptions about constructing items with __new__ methods should > probably still be clarified; I just replaced references to __new__ with > references to the corresponding classes. > > - What to do with the material from numpy.doc.* should be decided, as the > text there doesn't look like it should go into a reference manual. > > Some questions: > > - Is this good enough to go into Numpy SVN at some point? > > Or should we redo it and base the work closer to the original > "Guide to Numpy"? > > - Does it build for you? > > (I'd recommend using the development 0.5 version of Sphinx, so that you > get the nifty Inter-Sphinx links to the Python documentation.) > > We are unfortunately beating the Sphinx with a big stick to make it > place the documentation of each function or class into a separate file, > and to convert the Numpy docstring format to something the Sphinx can > fathom. :) > > There's also some magic in place to make toctrees:: of function listings > more pleasant to the eye. > > Any comments of what should be improved are welcome. (Even better: clone > the bzr branch, make the changes yourself, and put the result somewhere > available! E.g. as a bzr bundle or a branch on the launchpad.) I think it looks excellent. It'd be cool if all the docs could finally be at one place, instead of scattered all over the wiki. So for me, any form in sphinx is ok. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy08] Documentation BoF
On Fri, Aug 22, 2008 at 10:26 AM, Stéfan van der Walt <[EMAIL PROTECTED]> wrote: > Hi all, > > This is my personal recollection of the documentation BoF. Feel free to > comment or correct the text below. > > Regards > Stéfan > > > Summary of the Documentation Birds-of-a-Feather Session > === > > Topics proposed for discussion > -- > > - Path towards getting SciPy documented using the online documentation > editor > - Intersphinx: linking Sphinx-using projects > - Culture / methodology: what do we expect from code accepted into > NumPy/SciPy? > - Tools for reading documentation (Help 2.0) > - The complete SciPy book -- how to get there > - NumPy Documentation Standard: is it useful beyond NumPy? > > Conclusions reached > --- > > Documentation format > > - Question: could we standardise the descriptions of data-types between docs > of different projects? They should overlap to some extent. > > - If we could agree on some common standard (or at least a common subset of > a format) across projects, IPython could parse and display docstrings with > special markup. > > - Matplotlib should be involved in this conversation. They have already > written a vast amount of documentation, and their needs may be different to > those of the SciPy projects. > > Formal requirements > ``` > For functions going into NumPy and SciPy, we have a certain minimum > documentation requirements. However, we'd rather take a somewhat liberal, > open approach, and accept code with somwhat inferior docstrings, working > with the author to improve them. This way, we remain accessible as a > community, while striving to improve the quality of the code base. > > Marketing SciPy > ``` > The current entry point for SciPy on the web (http://www.scipy.org) may be > confusing to beginners. The web page should be redesigned, in a similar > fashion as the new Sage and code.enthought.com pages, linking to the messy > wiki behind it in a consistent fashion. Joe Harrington may be able to hire > someone to work on this important aspect of SciPy marketing. > > Cross-project documentation searching > ` > Ideally, one should be able to perform a documentation search across several > projects. In order to achieve this, we should negotiate a common convention > for installing Sphinx-generated documentation into a standard location. A > javascript-backed webpage can then be provided to do a search, using the > json indices, or some other means. +1 Currently sphinx can't handle scipy docstrings, can it? It didn't for me at least. It'd be nice if whatever format you agre upon, could work with sphinx's autodoc. Also I am very interested in your web based editing of docstrings, so that people can just fix and improve the docstrings easily. I can imagine something like In [1]: e.series??? in ipython and it would fireup a browser pointing to the docstring and one would just fix it. Since you did most of the work already, maybe the above won't be that hard to do either. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible new multiplication operators for Python
On Tue, Aug 19, 2008 at 1:57 PM, Alan G Isaac <[EMAIL PROTECTED]> wrote: >> On Sun, Aug 17, 2008 at 04:28:55PM -0400, Alan G Isaac wrote: >>> That said, what kind of problems do you have in mind? > > Gael Varoquaux wrote: >> wht I am most worried about is not being able to enter the >> symbol, because I am in an editor I don't know, and the >> symbol is not on my keyboard. This has happened to me more >> than once with unicode. > > Well anyone who travels ends up dealing with strange > keyboards, I suppose, but I would hate for the possibility > that someone is in the odd situation of this happening > while not having access to the net (and thus to Vim or emacs) > dictate the decision on this. That would just not be > very forward looking, especially when one can *in this > case* always use ``dot`` instead. I absolutely agree with Gael here. Using anything besides lower (<128) ascii is imho not wise. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1)
On Tue, Aug 19, 2008 at 1:06 AM, Christian Heimes <[EMAIL PROTECTED]> wrote: > Andrew Dalke wrote: >> When would this "with float ... " considered valid? > > [long posting] > > Oh h... what have I done ... *g* > > Slow down, please. For now there are no concrete plans what-so-ever to > implement the feature in the near future. Some developers have expressed > their interest in a way to alter the resulting type of a literal. It was > my attention to show you, that we have discussed the idea, too. > > Now for the "with type as from import" syntax. I came up with the syntax > idea about an hour ago. I tried to come up with some nice syntax that > reuses existing keywords. IMHO it has a nice ring. Other possibilities I > came up with: > > def float as factory > def float as from module import factory > with float yield factory > with float yield from module import factory > > After some careful thinking I'm in favor of "with ... yield ...". It's > less ambiguous and can't be mistaken for "with open(filename) as fh". > > The ideas needs a good PEP. You are definitely up to something. You also > came up with a list of possible issues and corner cases. Are you > interested in pursuing the proposal? *wink* Are we able to provide an actual patch to Python that implements this? If so, then I am. Imho the proposal should come with an actual patch, otherwise it's difficult to judge it. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1)
Hi Christian, On Mon, Aug 18, 2008 at 11:22 PM, Christian Heimes <[EMAIL PROTECTED]> wrote: > Ondrej Certik wrote: > > Ok, in the current state, you don't know either what's going to >> happen. If you write >> >> In [1]: x/2*3/4 >> >> you have no idea what the result is going to be, you need to analyze >> x.__div__() and start from there. But if you write >> >> In [2]: 1/2*3/4 >> >> currently you know it will be 0. But imho you could as well analyze >> the global __mul__ (or global __int__, depending on how this would be >> technically implemented) to see what's going to happen. >> >> I mean what is the difference between [1] and [2]? > > Andrew has already pointed it out very well. I like to comment on your > proposal from the perspective of a Python core developer as well as the > perspective of somebody who has worked with Guido for more than a year. > > I'd bet my life that Guido is never every going to allow it. The core > types are fundemental to the Python interpreter. Even the possibility of > pluggable type methods would make the implementation slower, more > fragile and much more complicated. We'd have to remove several speed > tricks and special cases for e.g. ints and replace them with slower > implementations. > > But don't give up hope yet! During the alpha phase of Python 3.0 and the > revamping of the decimal module, some core developers had an even better > idea. We were discussing the possibility of making decimals the default > for float literals. The idea was rejected eventually but it gave birth > to yet another idea. What about making the *result* of a literal > pluggable? The Python creates a float for the literal 1.0. Some header > in a module could replace the default target 'float' with e.g. > decimal.Decimal. > > Example syntax (rough idea): > > >>> type(1.0) > > >>> with float as from decimal import Decimal > >>> type(1.0) > > > Wouldn't that solve your general problem more elegant without breaking > other modules? It absolutely would. Thanks very much for the email. How is your proposal (redefine literals) different to just saying to Python -- hey, just call my class when someone writes "1", e.g. proposition 2) from my first email? Or am I missing something. I agree with the technical reasonings, why some particular solution is not good. I.e. I didn't do any proposal, I am just trying to find a way, so that we don't have to always type In [3]: Integer(1)/2 * x sometimes, but In [4]: x/2 some other times. If you know what I mean. Both do the same thing, but [1] is very annoying to write and a source of common mistakes that people do with SymPy, it simply returns 0. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1)
On Mon, Aug 18, 2008 at 10:45 PM, Andrew Dalke <[EMAIL PROTECTED]> wrote: > On Aug 18, 2008, at 10:01 PM, Ondrej Certik wrote: > >> with Andrew permission, I am starting a new thread, where our >> discussion is ontopic. :) > > Though I want to point out that without specific proposals > of how the implementation might look, this thread will > not go anywhere as it will be too distant from usable code. > > I sent examples to show how such a system might look, as > the basis for getting a feel if it was practical. I do > not think my examples are practical, but they were meant > as an example of how such a proposal might look. > > Since I know that the Python implementation will not change > to support per-module or per-scope redefinitions for "1+2" > and builtin object constructors, the only feasible mechanism > is through some sort of alternate grammar that compiles to > either Python or directly to the Python virtual machine. > > One such was is through import hooks. > >> Yes, this is a mess, this is just like preparsing. Well, not like -- >> this is preparsing. >> > > It's not preparsing. It's parsing. There's no pre about it. > It's not a macro language. My ply4python tutorial compiles > various Python-like languages to the Python virtual machine > bytecode. > > >> I mean what is the difference between [1] and [2]? > > I want to see how you would extend Python to support such > a mechanism before I worried about how to interpret it. > > Or in other words, the difference between [1] and [2] > is that [2] can be fully evaluated through simple static > analysis, while [1] cannot. > > BTW, this is unexpected. Python does constant folding > of that expression, but only with specific settings. > > >>> import dis > >>> def f(): > ... print 1/2*3/4 > ... > >>> dis.dis(f) > 2 0 LOAD_CONST 1 (1) > 3 LOAD_CONST 2 (2) > 6 BINARY_DIVIDE > 7 LOAD_CONST 3 (3) > 10 BINARY_MULTIPLY > 11 LOAD_CONST 4 (4) > 14 BINARY_DIVIDE > 15 PRINT_ITEM > 16 PRINT_NEWLINE > 17 LOAD_CONST 0 (None) > 20 RETURN_VALUE > >>> > >>> from __future__ import division > >>> def f(): > ... print 1/2*3/4 > ... > >>> dis.dis(f) > 2 0 LOAD_CONST 7 (0.375) > 3 PRINT_ITEM > 4 PRINT_NEWLINE > 5 LOAD_CONST 0 (None) > 8 RETURN_VALUE > >>> > > > > The only way I can see to do what you want requires > multimethods, which don't currently exist in Python > except as third-party extensions. The one I know about, > from Philip J. Eby, works on a global-level, not module > level, because of how registration happens, so it does > not support what you would like. One way to fix that would be to use the trick that Travis Oliphant told me at EuroSciPy -- hack the while (or if) statement and do the preparsing in there. So clearly the language seems to support that, so imho it could be made more official. E.g. you would write: while sympy: e = 1/2 and "e" would be Integer(1)/Integer(2). But anyway, it's kind of hackish and I don't know what to say more about it, besides what I already said. The problem is that I don't have time to dig more into Python internals and without it it seems I cannot provide more constructive answer besides I want some way to avoid Python reducing 1/2 to 0. Generally I believe that if there is will, it can always be done. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] global overloading of 1+1 -> MyClass(1, 1)
Hi, with Andrew's permission, I am starting a new thread, where our discussion is ontopic. :) My original question was, that I would like to override 1+1 to return MyClass(1, 1) or something. Robert said it would break other libraries and Andrew said this: On Mon, Aug 18, 2008 at 9:23 PM, Andrew Dalke <[EMAIL PROTECTED]> wrote: >> There are basically two possible options: >> >> 1) global overloading of + >> 2) allow to tell Python that "1" is not its int, but ours Integer. >> >> BTW, Sage just preparses Python for exactly this reason and >> substitutes all numbers like 1 with Integer(1). This really sucks >> imho. > > How would you like to do it? Any solution I can think of > would cause huge problems. For example, being able to change > int.__add__ would cause systemic problems throughout the > code base and would prevent, or very much hinder, migration > of Python code to C. Agree. > > Changes to support a different constructor for basic types, > on a per-module basis has its own problems. When are the > new types specified? In the module itself or by the code > which imports the module? Can the definitions be changed > on a scope-by-scope level? > > Now, I can imagine a Python with builtin multimethod > dispatch defined via static scoping, so that > > def __add__(left:int, right:int): > return __builtin__.__add__(__builtin__.__add__(left, right), 2) > > def __int__(s): > return Decimal(s) > > might work, but even here there's a problem because the "2" in > __add__ gets replaced by a Decimal, when I really want it to > be an integer. Not getting why is "2" replaced by Decimal if you don't want to? If you don't want it, you can just reimplement __int__(s). This would of course be only active in the module where it was defined. > > So I don't see how you can do it in the context of Python. > In the context of a very different programming language, > sure, it's possible. > > >> But that's what Sage is doing, they just preparse the input. But in >> SymPy we'd like to use it as a regular Python library in users >> programs, so imho preparsing, or custom compiling using your package >> is not really an option, is it? > > Use a different extension, like ".sympy" and an import hook > which does the .sympy -> .pyc conversion. > > But it's not one I recommend for real code, at least not > without a lot of personal experience to see if it's worthwhile > in the face of the downsides. Yes, this is a mess, this is just like preparsing. Well, not like -- this is preparsing. Ok, in the current state, you don't know either what's going to happen. If you write In [1]: x/2*3/4 you have no idea what the result is going to be, you need to analyze x.__div__() and start from there. But if you write In [2]: 1/2*3/4 currently you know it will be 0. But imho you could as well analyze the global __mul__ (or global __int__, depending on how this would be technically implemented) to see what's going to happen. I mean what is the difference between [1] and [2]? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible new multiplication operators for Python
On Mon, Aug 18, 2008 at 1:50 AM, Andrew Dalke <[EMAIL PROTECTED]> wrote: > On Aug 18, 2008, at 12:00 AM, Ondrej Certik wrote: >> There is some inconsistency though, for example one can override A() + >> A(), but one cannot override 1 + 1. This could (should) be fixed >> somehow. > > That will never, ever change in Python. There's no benefit > to being able to redefine int.__add__ and doing so will break > entirely too many assumptions. > > Here's one assumption - the C implementation does some > simple constant folding: > > >>> def f(): > ... print 1+1 > ... > >>> import dis > >>> dis.dis(f) > 2 0 LOAD_CONST 2 (2) > 3 PRINT_ITEM > 4 PRINT_NEWLINE > 5 LOAD_CONST 0 (None) > 8 RETURN_VALUE > >>> > > With what you want that's not possible. > > > Just think of the implementation difficulty. Are changes on the > per-module or per-scope or per-thread level? And performance > would be lousy (or require deep inferencing analysis) because > every operation in C would need to go through the Python API > just in case some fundamental definition like this was changed. > > Such a language is possible. I wouldn't be surprised if > you could do it in Smalltalk and mostly do it in Ruby. But > the general belief is that good design follows the > "open/closed principle": > > "software entities (classes, modules, functions, etc.) > should be open for extension, but closed for modification" > - Bertrand Meyer, quoted by > http://en.wikipedia.org/wiki/Open/closed_principle > > In Python, all types are closed for modification, and > while classes are open for modification it's highly frowned > upon to do so. The name "monkey-patch" sounds somewhat > derisive for a reason. Yeah, I understand the reasoning. My reason is that sometimes you want to do something else on 1/2 rather than to get 0, or 0.500. I would like to get my own class called, but if it's again Python, then I am perfectly happy with Python as it is now. No changes needed. Anyway, this is off topic. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible new multiplication operators for Python
On Sun, Aug 17, 2008 at 7:03 AM, Fernando Perez <[EMAIL PROTECTED]> wrote: > Hi all, > > [ please keep all replies to this only on the numpy list. I'm cc'ing > the scipy ones to make others aware of the topic, but do NOT reply on > those lists so we can have an organized thread for future reference] > > In the Python-dev mailing lists, there were recently two threads > regarding the possibility of adding to the language new multiplication > operators (amongst others). This would allow one to define things > like an element-wise and a matrix product for numpy arrays, for > example: > > http://mail.python.org/pipermail/python-dev/2008-July/081508.html > http://mail.python.org/pipermail/python-dev/2008-July/081551.html > > It turns out that there's an old pep on this issue: > > http://www.python.org/dev/peps/pep-0225/ > > which hasn't been ruled out, simply postponed. At this point it seems > that there is room for some discussion, and obviously the input of the > numpy/scipy crowd would be very welcome. I volunteered to host a BOF > next week at scipy so we could collect feedback from those present, > but it's important that those NOT present at the conference can > equally voice their ideas/opinions. > > So I wanted to open this thread here to collect feedback. We'll then > try to have the bof next week at the conference, and I'll summarize > everything for python-dev. Obviously this doesn't mean that we'll get > any changes in, but at least there's interest in discussing a topic > that has been dear to everyone here. I like that Python is so easy to learn, so I hope no more operators and definitely not unicode operators will be added. And if, then only if they are really needed, which I think they are not. What I like on Python is that I can remember all it's syntax in my memory. The more operators, the more difficult this will be. There is some inconsistency though, for example one can override A() + A(), but one cannot override 1 + 1. This could (should) be fixed somehow. But apart from that, I very much like Python as it is now and I hope it will not become a bloated language. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] "import numpy" is slow
On Thu, Jul 31, 2008 at 10:02 PM, Robert Kern <[EMAIL PROTECTED]> wrote:
> On Thu, Jul 31, 2008 at 05:43, Andrew Dalke <[EMAIL PROTECTED]> wrote:
>> On Jul 31, 2008, at 12:03 PM, Robert Kern wrote:
>
>>> But you still can't remove them since they are being used inside
>>> numerictypes. That's why I labeled them "internal utility functions"
>>> instead of leaving them with minimal docstrings such that you would
>>> have to guess.
>>
>> My proposal is to replace that code with a table mapping
>> the type name to the uppercase/lowercase/capitalized forms,
>> thus eliminating the (small) amount of time needed to
>> import string.
>>
>> It makes adding new types slightly more difficult.
>>
>> I know it's a tradeoff.
>
> Probably not a bad one. Write up the patch, and then we'll see how
> much it affects the import time.
>
> I would much rather that we discuss concrete changes like this rather
> than rehash the justifications of old decisions. Regardless of the
> merits about the old decisions (and I agreed with your position at the
> time), it's a pointless and irrelevant conversation. The decisions
> were made, and now we have a user base to whom we have promised not to
> break their code so egregiously again. The relevant conversation is
> what changes we can make now.
>
> Some general guidelines:
>
> 1) Everything exposed by "from numpy import *" still needs to work.
> a) The layout of everything under numpy.core is an implementation detail.
> b) _underscored functions and explicitly labeled internal functions
> can probably be modified.
> c) Ask about specific functions when in doubt.
>
> 2) The improvement in import times should be substantial. Feel free to
> bundle up the optimizations for consideration.
>
> 3) Moving imports from module-level down into the functions where they
> are used is generally okay if we get a reasonable win from it. The
> local imports should be commented, explaining that they are made local
> in order to improve the import times.
>
> 4) __import__ hacks are off the table.
>
> 5) Proxy objects ... I would really like to avoid proxy objects. They
> have caused fragility in the past.
>
> 6) I'm not a fan of having environment variables control the way numpy
> gets imported, but I'm willing to consider it. For example, I might go
> for having proxy objects for linalg et al. *only* if a particular
> environment variable were set. But there had better be a very large
> improvement in import times.
I just want to say that I agree with Andrew that slow imports just
suck. But it's not really that bad, for example on my system:
In [1]: %time import numpy
CPU times: user 0.11 s, sys: 0.01 s, total: 0.12 s
Wall time: 0.12 s
so that's ok. For comparison:
In [1]: %time import sympy
CPU times: user 0.12 s, sys: 0.02 s, total: 0.14 s
Wall time: 0.14 s
But I am still unhappy about it, I'd like if the package could import
much faster, because it adds up, when you need to import 7 packages
like that, it's suddenly 1s and that's just too much.
But of course everything within the constrains that Robert has
outlined. From the theoretical point of view, I don't understand why
python cannot just import numpy (or any other package) immediatelly,
and only at the moment the user actually access something, to import
it in real. Mercurial uses a lazy import module, that does exactly
this. Maybe that's an option?
Look into mercurial/demandimport.py.
Use it like this:
In [1]: import demandimport
In [2]: demandimport.enable()
In [3]: %time import numpy
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s
That's pretty good, huh? :)
Unfortunately, numpy cannot work with lazy import (yet):
In [5]: %time from numpy import array
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (17, 0))
---
AttributeErrorTraceback (most recent call last)
[skip]
/usr/lib/python2.5/site-packages/numpy/lib/index_tricks.py in ()
14 import function_base
15 import numpy.core.defmatrix as matrix
---> 16 makemat = matrix.matrix
17
18 # contributed by Stefan van der Walt
/home/ondra/ext/sympy/demandimport.pyc in __getattribute__(self, attr)
73 return object.__getattribute__(self, attr)
74 self._load()
---> 75 return getattr(self._module, attr)
76 def __setattr__(self, attr, val):
77 self._load()
AttributeError: 'module' object has no attribute 'matrix'
BTW, neither can SymPy. However, maybe it shows some possibilities and
maybe it's possible to fix numpy to work with such a lazy import.
On the other hand, I can imagine it can bring a lot more troubles, so
it should probably only be optional.
Ondrej
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
h
Re: [Numpy-discussion] Debian: numpy not building _dotblas.so
On Tue, Jul 8, 2008 at 10:19 PM, Robert Kern <[EMAIL PROTECTED]> wrote: > On Tue, Jul 8, 2008 at 14:47, Ondrej Certik <[EMAIL PROTECTED]> wrote: >> On Tue, Jul 8, 2008 at 9:15 PM, Robert Kern <[EMAIL PROTECTED]> wrote: >>> On Tue, Jul 8, 2008 at 08:06, Ondrej Certik <[EMAIL PROTECTED]> wrote: >>>> On Tue, Jul 8, 2008 at 11:48 AM, Tiziano Zito <[EMAIL PROTECTED]> wrote: >>>>> Hi numpy-devs, I was the one reporting the original bug about missing >>>>> ATLAS >>>>> support in the debian lenny python-numpy package. AFAICT the source >>>>> python-numpy package in etch (numpy version 1.0.1) does not require >>>>> atlas to build >>>>> _dotblas.c, only lapack is needed. If you install the resulting binary >>>>> package on a >>>>> system where ATLAS is present, ATLAS libraries are used instead of plain >>>>> lapack. >>>>> So basically it was already working before the check for ATLAS was >>>>> introduced into >>>>> the numpy building system. Why should ATLAS now be required? >>>>> >>>>>> It's not as trivial as just reverting that changeset, though. >>>>> why is that? I mean, it was *working* before... >>>> >>>> So just removing the two lines from numpy seems to fix the problem in >>>> Debian. So far all tests seem to run both on i386 and amd64, both with >>>> and without atlas packages installed. And it is indeed faster with the >>>> altas packages instaled, yet it doesn't need them to build. I think >>>> that's what we want, no? >>> >>> Can you give me more details? >> >> Sure. :) >> >>> Was the binary built on a machine with >>> an absent ATLAS? >> >> Yes, the binary is always built on a machine with an absent atlas, as >> the package is build-conflicting with atlas. >> >>> Show me the output of ldd on _dotblas.so with both >>> ATLAS installed and not. Can you import numpy.core._dotblas explicitly >>> under both? >> >> ATLAS installed: >> >> [EMAIL PROTECTED]:~/debian$ ldd >> /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so >>linux-gate.so.1 => (0xb7fba000) >>libblas.so.3gf => /usr/lib/atlas/libblas.so.3gf (0xb7c19000) >>libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0xb7b67000) >>libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb7b4) >>libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7b33000) >>libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb79d8000) >>/lib/ld-linux.so.2 (0xb7fbb000) >> [EMAIL PROTECTED]:~/debian$ python >> Python 2.5.2 (r252:60911, Jun 25 2008, 17:58:32) >> [GCC 4.3.1] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import numpy.core._dotblas >>>>> >> >> >> ATLAS not installed: >> >> [EMAIL PROTECTED]:~/debian$ ldd >> /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so >>linux-gate.so.1 => (0xb7f2f000) >>libblas.so.3gf => /usr/lib/libblas.so.3gf (0xb7e82000) >>libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0xb7dd) >>libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb7da9000) >>libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7d9c000) >>libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb7c41000) >>/lib/ld-linux.so.2 (0xb7f3) >> [EMAIL PROTECTED]:~/debian$ python >> Python 2.5.2 (r252:60911, Jun 25 2008, 17:58:32) >> [GCC 4.3.1] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import numpy.core._dotblas >>>>> > > Okay, it turns out that libblas on Ubuntu (and I'm guessing Debian) > includes the CBLAS interface. > > $ nm /usr/lib/libblas.a | grep "T cblas_" > T cblas_caxpy > T cblas_ccopy > ... > > This is specific to Debian and its derivatives. Not all libblas's have > this. So I stand by my statement that just reverting the change is not > acceptable. We need a real check for the CBLAS interface. In the Right. > meantime, the Debian package maintainer can patch the file to remove > that check during the build for Debian systems. Yes, I just did that. Thanks for the clarification. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Debian: numpy not building _dotblas.so
On Tue, Jul 8, 2008 at 9:15 PM, Robert Kern <[EMAIL PROTECTED]> wrote: > On Tue, Jul 8, 2008 at 08:06, Ondrej Certik <[EMAIL PROTECTED]> wrote: >> On Tue, Jul 8, 2008 at 11:48 AM, Tiziano Zito <[EMAIL PROTECTED]> wrote: >>> Hi numpy-devs, I was the one reporting the original bug about missing ATLAS >>> support in the debian lenny python-numpy package. AFAICT the source >>> python-numpy package in etch (numpy version 1.0.1) does not require >>> atlas to build >>> _dotblas.c, only lapack is needed. If you install the resulting binary >>> package on a >>> system where ATLAS is present, ATLAS libraries are used instead of plain >>> lapack. >>> So basically it was already working before the check for ATLAS was >>> introduced into >>> the numpy building system. Why should ATLAS now be required? >>> >>>> It's not as trivial as just reverting that changeset, though. >>> why is that? I mean, it was *working* before... >> >> So just removing the two lines from numpy seems to fix the problem in >> Debian. So far all tests seem to run both on i386 and amd64, both with >> and without atlas packages installed. And it is indeed faster with the >> altas packages instaled, yet it doesn't need them to build. I think >> that's what we want, no? > > Can you give me more details? Sure. :) > Was the binary built on a machine with > an absent ATLAS? Yes, the binary is always built on a machine with an absent atlas, as the package is build-conflicting with atlas. > Show me the output of ldd on _dotblas.so with both > ATLAS installed and not. Can you import numpy.core._dotblas explicitly > under both? ATLAS installed: [EMAIL PROTECTED]:~/debian$ ldd /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so linux-gate.so.1 => (0xb7fba000) libblas.so.3gf => /usr/lib/atlas/libblas.so.3gf (0xb7c19000) libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0xb7b67000) libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb7b4) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7b33000) libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb79d8000) /lib/ld-linux.so.2 (0xb7fbb000) [EMAIL PROTECTED]:~/debian$ python Python 2.5.2 (r252:60911, Jun 25 2008, 17:58:32) [GCC 4.3.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy.core._dotblas >>> ATLAS not installed: [EMAIL PROTECTED]:~/debian$ ldd /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so linux-gate.so.1 => (0xb7f2f000) libblas.so.3gf => /usr/lib/libblas.so.3gf (0xb7e82000) libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0xb7dd) libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb7da9000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7d9c000) libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb7c41000) /lib/ld-linux.so.2 (0xb7f3) [EMAIL PROTECTED]:~/debian$ python Python 2.5.2 (r252:60911, Jun 25 2008, 17:58:32) [GCC 4.3.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy.core._dotblas >>> Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Debian: numpy not building _dotblas.so
On Tue, Jul 8, 2008 at 11:48 AM, Tiziano Zito <[EMAIL PROTECTED]> wrote: > Hi numpy-devs, I was the one reporting the original bug about missing ATLAS > support in the debian lenny python-numpy package. AFAICT the source > python-numpy package in etch (numpy version 1.0.1) does not require > atlas to build > _dotblas.c, only lapack is needed. If you install the resulting binary > package on a > system where ATLAS is present, ATLAS libraries are used instead of plain > lapack. > So basically it was already working before the check for ATLAS was > introduced into > the numpy building system. Why should ATLAS now be required? > >> It's not as trivial as just reverting that changeset, though. > why is that? I mean, it was *working* before... So just removing the two lines from numpy seems to fix the problem in Debian. So far all tests seem to run both on i386 and amd64, both with and without atlas packages installed. And it is indeed faster with the altas packages instaled, yet it doesn't need them to build. I think that's what we want, no? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Debian: numpy not building _dotblas.so
Hi, we have this problem in Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=489726 The problem is that numpy should not depend on atlas unconditionally, yet it should allow it for users that have it. I am not an expert in blas/lapack/atlas and it's Debian packaging much (I know some people say that atlas packaging in Debian is not very good, actually pretty bad), so I am just forwarding the question here. The problem is with this patch: http://projects.scipy.org/scipy/numpy/changeset/3854 and the question that we have is: I'd like to know, if the code was changed to only work with atlas, or if was never working. if it's the latter, then we should use atlas Matthias, Tiziano, feel free to clarify this more. See the above Debian bug for more information and background. Thanks, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] array vs matrix
On Sun, Jun 8, 2008 at 3:54 AM, Anne Archibald <[EMAIL PROTECTED]> wrote: > 2008/6/7 Robert Kern <[EMAIL PROTECTED]>: >> On Sat, Jun 7, 2008 at 14:37, Ondrej Certik <[EMAIL PROTECTED]> wrote: >>> Hi, >>> >>> what is the current plan with array and matrix with regard of calculating >>> >>> sin(A) >>> >>> ? I.e. elementwise vs sin(A) = Q*sin(D)*Q^T? Is the current approach >>> (elementwise for array and Q*sin(D)*Q^T for matrix) the way to go? >> >> I don't believe we intend to make numpy.matrix any more featureful. I >> don't think it's a good idea for you to base sympy.Matrix's >> capabilities in lock-step with numpy.matrix, though. There are very >> different constraints at work. Please, do what you think is best for >> sympy.Matrix. > > Let me elaborate somewhat: > > We recently ran across some painful quirks in numpy's handling of > matrices, and they spawned massive discussion. As it stands now, there > is significant interest in reimplementing the matrix object from > scratch, with different behaviour. So emulating its current behaviour > is not a win. > > For consistency, it makes a certain amount of sense to have sin(A) > compute a matrix sine, since A**n computes a matrix power. But looking > at the matrix exponential, I see that we have several implementations, > none of which is satisfactory for all matrices. I would expect the > matrix sine to be similar - particularly when faced with complex > matrices - so perhaps needing an explicit matrix sine is a good thing? > > Also worth noting is that this problem can be evaded with namespaces; > matrix sin could be scipy.matrixmath.sin, abbreviated perhaps to > mm.sin, as opposed to np.sin. I see. Thanks Robert and Anne for the replies. That's all I needed to know. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy 1.1: import numpy fails in Debian
On Mon, Jun 9, 2008 at 6:20 PM, Ondrej Certik <[EMAIL PROTECTED]> wrote: > Hi, > > I tried to package numpy 1.1 and upload it to Debian, all worked fine, > I installed the package and: > > In [1]: import numpy > --- > ImportError Traceback (most recent call last) > > /home/ondra/debian/packages/numpy/ in () > > /usr/lib/python2.5/site-packages/numpy/__init__.py in () >105 import random >106 import ctypeslib > --> 107 import ma >global ma = undefined >108 >109 # Make these accessible from numpy name-space > > ImportError: No module named ma > > > > So I tried to investigate where the problem is, but I didn't figure it > out. I am sending the build log. If anyone knows what's wrong, it'd be > very helpful. > > The only thing I figured out so far is that the "ma" package gets > installed into: > > debian/tmp/usr/lib/python2.5/site-packages/numpy/ma > > but it is not copied into the directory: > > debian/python-numpy/usr/share/pyshared/numpy/ > > from which the package is made. So it's maybe a bug in some Debian > package, rather than in numpy itself. But maybe some numpy developers > know what is so different on the "ma" package (which as I understand > was added in 1.1). Andrew Straw fixed the problem, so the new numpy package is in incoming: http://incoming.debian.org/ and should be on Debian mirrors tomorrow. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] array vs matrix
Hi, what is the current plan with array and matrix with regard of calculating sin(A) ? I.e. elementwise vs sin(A) = Q*sin(D)*Q^T? Is the current approach (elementwise for array and Q*sin(D)*Q^T for matrix) the way to go? We are solving the same problem in SymPy: http://groups.google.com/group/sympy/browse_thread/thread/6cfe66b012b24948 so I'd like to have the exact same behavior as numpy has, so that we are compatible. But I was wondering if the current status is ok, or if many numpy developers (or users?) would like this to be changed. I think a clean solution is to treat matrices as mathematical objects, i.e. A*B and sin(A) being matrix multiplication and matrix sin respectively, while leaving the array for elementwise operations. But above all, I'd like to be compatible with numpy, otherwise it will be a mess. Thanks for any feedback, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ticket #798: `piecewise` exposes raw memory
On Wed, May 21, 2008 at 11:53 AM, Ondrej Certik <[EMAIL PROTECTED]> wrote:
> On Wed, May 21, 2008 at 11:30 AM, Stéfan van der Walt <[EMAIL PROTECTED]>
> wrote:
>> Referring to
>> http://scipy.org/scipy/numpy/ticket/798
>>
>> `piecewise` uses `empty` to allocate output memory. If the conditions
>> do not sufficiently cover the output, then raw memory is returned,
>> e.g.,
>>
>> {{{
>> import numpy as np
>> np.piecewise([0,1,2],[True,False,False],[1])
>> }}}
>>
>> A patch which addresses the issue is available here for review:
>>
>> http://codereview.appspot.com/1105
>>
>> Documentation is being updated on the wiki.
>
> I'd like to invite everyone to take part in the review. It's fun, it's
> just talking, no coding. :)
Thanks Robert for taking part in the review:
http://codereview.appspot.com/1105/diff/22/122#newcode566
That's the way to go.
Ondrej
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Ticket #798: `piecewise` exposes raw memory
On Wed, May 21, 2008 at 11:30 AM, Stéfan van der Walt <[EMAIL PROTECTED]> wrote:
> Referring to
> http://scipy.org/scipy/numpy/ticket/798
>
> `piecewise` uses `empty` to allocate output memory. If the conditions
> do not sufficiently cover the output, then raw memory is returned,
> e.g.,
>
> {{{
> import numpy as np
> np.piecewise([0,1,2],[True,False,False],[1])
> }}}
>
> A patch which addresses the issue is available here for review:
>
> http://codereview.appspot.com/1105
>
> Documentation is being updated on the wiki.
I'd like to invite everyone to take part in the review. It's fun, it's
just talking, no coding. :)
Ondrej
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] let's use patch review
On Thu, May 15, 2008 at 3:08 PM, David Huard <[EMAIL PROTECTED]> wrote: > 2008/5/14 David Cournapeau <[EMAIL PROTECTED]>: >> >> On Wed, 2008-05-14 at 13:58 -1000, Eric Firing wrote: >> > >> > What does that mean? How does one know when there is a consensus? >> >> There can be a system to make this automatic. For example, the code is >> never commited directly to svn, but to a gatekeeper, and people vote by >> an email command to say if they want the patch in; when the total number >> of votes is above some threshold, the gatekeeper commit the patch. > > There is about 5 commits/day, I don't think it's a good idea to wait for a > vote on each one of them. Me neither. I think just one reviewer is enough. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] let's use patch review
On Thu, May 15, 2008 at 1:58 AM, Eric Firing <[EMAIL PROTECTED]> wrote: > Ondrej Certik wrote: >> Hi, >> >> I read the recent flamebate about unittests, formal procedures for a >> commit etc. and it was amusing. :) >> I think Stefan is right about the unit tests. I also think that Travis >> is right that there is no formal procedure that can assure what we >> want. >> >> I think that a solution is a patch review. Every big/succesful project >> does it. And the workflow as I see it is this: > > Are you sure numpy is big enough that a formal mechanism is needed--for > everyone? It makes good sense for my (rare) patches to be reviewed, but > shouldn't some of the core developers be allowed to simply get on with > it? As it is, my patches can easily be reviewed because I don't have > commit access. It's not me who should make this decision, as I have contributed in total maybe 1 patch to numpy. I am just suggesting it as a possibility. The numpy developers are free to choose the workflow that suits them best. But if you are asking for my own opinion -- yes, I think all code should be reviewed, including core developers, that's for example what Sage does (or what we do in SymPy), because that brings other people to comment on it, to help find bugs, to get familiar with it and also everyone involved learns something from it. Simply google why patch review is useful to get arguments for doing it. > >> >> 1) Travis will fix a bug, and submit it to a patch review. If he is >> busy, that's the only thing he will do >> 2) Someone else reviews it. Stefan will be the one who will always >> point out missing tests. > > That we can agree on! > >> 3) There needs to be a common consensus that the patch is ok to go in. > > What does that mean? How does one know when there is a consensus? That everyone involved in the discussion agrees it should go in as it is. I am sure you can recognize very easily if there is not a concensus. > >> 4) when the patch is reviewed and ok to go in, anyone with a commit >> access will commit it. > > But it has to be a specific person in each case, not "anyone". Those, who have commit access are definitely not anyone. Only those, who have showed they can be trusted not to break things. > >> >> I think it's as simple as that. >> >> Sometimes no one has enought time to write a proper test, yet someone >> has a free minute to fix a bug. Then I think it's ok to put the code >> in, as I think it's good to fix a bug now. However, > > How does that fit with the workflow above? Does Travis commit the > bugfix, or not? Both is possible. Some projects require all patches to go through a review and I personally think it's a good idea. > >> the issue is definitely not closed and the bug is not fixed (!) until >> someone writes a proper test. I.e. putting code in that is not tested, >> however it doesn't break things, is imho ok, as it will not hurt >> anyone and it will temporarily fix a bug (but of course the code will >> be broken at some point in the future, if there is no test for it). > That is overstating the case; for 788, for example, no one in his right > mind would undo the one-line correction that Travis made. Chances are, > there will be all sorts of breakage and foulups and the revelation of > new bugs in the future--but not another instance that would be caught by > the test for 788. That's what the patch review is for -- people will comment on this and if a consencus is made that a test is not necessary, ok. >> > [...] >> http://codereview.appspot.com/953 >> >> and added some comments. So what do you think? > Looks like it could be useful. I replied to the comments. I haven't > read the docs, and I don't know what the next step is when a revision of > the patch is in order, as it is in this case. It seems only the owner of the issue (in this case me, because I uploaded your code) can add new patches to that issue. So simply start a new issue and upload it there. If there were more revisions from you, it would look like this: http://codereview.appspot.com/970 Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] let's use patch review
Hi, I read the recent flamebate about unittests, formal procedures for a commit etc. and it was amusing. :) I think Stefan is right about the unit tests. I also think that Travis is right that there is no formal procedure that can assure what we want. I think that a solution is a patch review. Every big/succesful project does it. And the workflow as I see it is this: 1) Travis will fix a bug, and submit it to a patch review. If he is busy, that's the only thing he will do 2) Someone else reviews it. Stefan will be the one who will always point out missing tests. 3) There needs to be a common consensus that the patch is ok to go in. 4) when the patch is reviewed and ok to go in, anyone with a commit access will commit it. I think it's as simple as that. Sometimes no one has enought time to write a proper test, yet someone has a free minute to fix a bug. Then I think it's ok to put the code in, as I think it's good to fix a bug now. However, the issue is definitely not closed and the bug is not fixed (!) until someone writes a proper test. I.e. putting code in that is not tested, however it doesn't break things, is imho ok, as it will not hurt anyone and it will temporarily fix a bug (but of course the code will be broken at some point in the future, if there is no test for it). Now, the problem is that all patch review tools sucks in some way. Currently the most promissing is the one from Guido here: http://code.google.com/p/rietveld/ it's opensource, you can run it on your server, or use it online here: http://codereview.appspot.com/ I suggest you to read the docs how to use it, I am still learning it. Also it works fine for svn, but not for Mercurial, so we are not using it in SymPy yet. So to also do some work besides just talk, I started with this issue: http://projects.scipy.org/scipy/numpy/ticket/788 and submitted the code (not my code though:) in there for a review here: http://codereview.appspot.com/953 and added some comments. So what do you think? Ondrej P.S. efiring, my comments are real questions to your patch. :) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy release
On Wed, Apr 23, 2008 at 4:56 PM, Sebastian Haase <[EMAIL PROTECTED]> wrote: > > On Wed, Apr 23, 2008 at 2:32 PM, Ondrej Certik <[EMAIL PROTECTED]> wrote: > > On Mon, Apr 14, 2008 at 12:06 PM, Ondrej Certik <[EMAIL PROTECTED]> wrote: > > > Hi Jarrod, > > > > > > any news with the 1.0.5? If you have same prerelease, I'd like to test > > > it. Debian has just moved from python2.4 to python2.5 yesterday, so > > > I'd like to test numpy in advance, I am sure there will be some issues > > > to fix. > > > > What is the plan with the release? There are some minor problems in > > the Debian package, some of which are fixed by the new release. > > I didn't fix those in Debian as I thought the new release is coming > > out. But if it's going to take let's say month or two, I'll fix the > > current Debian package now. > > > > The problem is, that it was noticed that non-backwards compatible > changes were committed into svn. > Most noticeably a complete rewrite of the "ma" (Masked Array) package was > done. > As a consequence it was decided, they no one would be interested in > "temporarily taking those changes out" "just to release 1.0.5". > > Instead the next release will be called 1.1 -- which is (only; rather > still) "mostly" compatible with 1.0.x > What used to be referred to a the 1.1 version, that can break more > stuff, to allow for a cleaner design, will now be 1.2 OK, so I think we are going to try to fix the package in Debian right now and not wait for the release. I think this uncertainty about releases and about backward compatibility is not good. The same about uncertainty where f2py is going to end up. But you know the drill "talk is cheap, show me the code", so I know I could step up and help Jarrod with the release, but unfortunately, I don't have time for it. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy release
On Mon, Apr 14, 2008 at 12:06 PM, Ondrej Certik <[EMAIL PROTECTED]> wrote: > Hi Jarrod, > > any news with the 1.0.5? If you have same prerelease, I'd like to test > it. Debian has just moved from python2.4 to python2.5 yesterday, so > I'd like to test numpy in advance, I am sure there will be some issues > to fix. What is the plan with the release? There are some minor problems in the Debian package, some of which are fixed by the new release. I didn't fix those in Debian as I thought the new release is coming out. But if it's going to take let's say month or two, I'll fix the current Debian package now. Thanks, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Py3K
On Wed, Apr 16, 2008 at 2:56 PM, Stéfan van der Walt <[EMAIL PROTECTED]> wrote: > On 16/04/2008, Fernando Perez <[EMAIL PROTECTED]> wrote: > > On Tue, Apr 15, 2008 at 1:27 PM, Charles R Harris > > > > Oh, and making the transition will be made a lot easier by having a > > > complete > > > set of tests. Getting the tests and documentation into good state might > be > > > the best focus for our 1.2 effort. > > > > > > +1 > > > > Bonus points for whoever adds coverage reporting to our test suite, so > > that we can get at least basic metrics of coverage by: > > > > - docstrings > > - doctests > > - plain tests > > I'd love to see that. In the meantime, here is a figleaf coverage report: > > http://mentat.za.net/numpy/coverage/ Wow, figleaf is really cool. I didn't know about this tool. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy release
Hi Jarrod, any news with the 1.0.5? If you have same prerelease, I'd like to test it. Debian has just moved from python2.4 to python2.5 yesterday, so I'd like to test numpy in advance, I am sure there will be some issues to fix. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] mercurial now has free hosting too
Hi, since there was so much discussion whether bzr or hg, Mercurial has now free hosting too: http://freehg.org/ also Mercurial 1.0 was finally released yesterday. Bzr has Launchpad, that's one of the (main) reasons ipython is investigating it, so I am still learning how to use bzr, but it's not really so much different. The only little annoying thing I discovered so far is that it feels slower than hg, on regular tasks, like "bzr st", "bzr pull", "bzr up", etc. Only a little bit, but still it creates the feeling in me, that something is missing -- it's like if you are used to a fast car and then you get into a slower car - even though it's fast enough to drive you to the shop, you still are missing something, at least I am. :) But generally I wanted to say, that I think bzr is a good choice too. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy/Cython Google Summer of Code project idea
On Thu, Mar 6, 2008 at 9:48 PM, Robin <[EMAIL PROTECTED]> wrote: > On Thu, Mar 6, 2008 at 6:15 PM, Fernando Perez <[EMAIL PROTECTED]> wrote: > > Any student interested in this should quickly respond on the list; > > such a project would likely be co-mentored by people on the Numpy and > > Cython teams, since it is likely to require expertise from both ends. > > Hello, > > I would like to register my keen interest in applying for Numpy/Scipy > GSoC project. I am a first year PhD student in Computational > Neuroscience and my undergraduate degree was in Mathematics. > > I have been using Numpy and Scipy for my PhD work for a few months now > and have been building up to trying to contribute something to the > project - I am keen to get more substantial real world programming > experience... The projects described involving Cython definitely > interest me, although I don't yet have a sufficient understanding of > the Python C-API and Pyrex/Cython to gauge how demanding they might > be. > > As a PhD student in the UK I don't have any official summer vacation, > so I wouldn't be able to work full time on the project (I also have a > continuation report due just before the GSoC final deadline which is a > bit annoying). However I currently work 20 hours per week part time > anyway, so I'm confident that I could replace that with GSoC and still > keep up with my studies. Just a note, that the usual commitment is 40 hours/week, i.e. a full time job. See e.g.: http://wiki.python.org/moin/SummerOfCode/Expectations Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy 1:1.0.4: numpy.average() returns the wrong result with weights
On Sun, Feb 24, 2008 at 5:41 AM, Albert Strasheim <[EMAIL PROTECTED]> wrote: > Hello, > > - Original Message - > From: "Robert Kern" <[EMAIL PROTECTED]> > To: "Discussion of Numerical Python" > Sent: Saturday, February 23, 2008 3:30 AM > Subject: Re: [Numpy-discussion] numpy 1:1.0.4: numpy.average() returns the > wrong result with weights > > > > > Ondrej Certik wrote: > >> I'll add it. I registered on the trac, as required, but I am still > >> denied, when filling my username and password when logging in. > >> How can I create an account? > > > > That should have done it. When you say you are "denied", exactly what > > happens? > > I've run into times when I've logged in and I get the unaltered front page > > again. Logging in again usually works. > > There is something strange going on. Logging in on > projects.scipy.org/scipy/numpy usually redirects you to > scipy.org/scipy/numpy, at which point you need to log in again. I tried it several times now - register -> login -> it doesn't accept the password. Then it redirects to scipy.org/scipy/numpy as you said. Then I went back to the original page and then the login worked. (But I did the same trick before and it didn't work). Anyway, it seems to be working now, thanks for the help. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy 1:1.0.4: numpy.average() returns the wrong result with weights
On Sat, Feb 23, 2008 at 2:10 AM, Travis E. Oliphant <[EMAIL PROTECTED]> wrote: > > Ondrej Certik wrote: > > Hi, > > > > more details in this bug report. > > > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=467095 > > > > The bug report offers a fix for this problem. It seems to me this is > > not fixed even in the latest svn. > > > > > Is there a ticket on the NumPy trac for this? We won't see it if there > isn't. Thanks for pointing us to the bug. I'll add it. I registered on the trac, as required, but I am still denied, when filling my username and password when logging in. How can I create an account? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy 1:1.0.4: numpy.average() returns the wrong result with weights
Hi, more details in this bug report. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=467095 The bug report offers a fix for this problem. It seems to me this is not fixed even in the latest svn. Thanks, Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy and C++ integration...
On Feb 5, 2008 11:52 AM, Gael Varoquaux <[EMAIL PROTECTED]> wrote: > On Tue, Feb 05, 2008 at 11:48:38AM +0100, Ondrej Certik wrote: > > I use Cython, mostly for the same reasons Gael is using ctypes - it's > > trivial. > > Actually, when I want to do something really trivial, I use > scipy.weave.inline ( see http://scipy.org/PerformancePython for an > example of scipy.weave.inline use). Of course it doesn't work when > linking to external libraries, but to accelerate a for loop, it's great. Yep. The power of Cython/Pyrex strategy is that you can work as I described here: http://ondrej.certik.cz/development/ Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy and C++ integration...
On Feb 5, 2008 11:23 AM, David Cournapeau <[EMAIL PROTECTED]> wrote: > Gael Varoquaux wrote: > > On Tue, Feb 05, 2008 at 09:15:29AM +0100, Sebastian Haase wrote: > > > >> Can ctypes do this ? > >> > > > > No. Ctypes is only a way of loading C (and not C++) libraries in Python. > > That makes it very simple, but not very powerful. > > > I would not call ctypes not very powerful :) For sure you cannot do the > same way as swig does, but you could imagine some automatic scheme to > solve Sebastian's problem. > > Typically, having a C wrapper automatically generated from the C++ > headers, you could use the ctypes code generator, and you have something > almost automatic (with a bit some boilerplate code in python, maybe). > > cheers, Also feel free to extend this wiki: http://wiki.cython.org/WrappingCorCpp I use Cython, mostly for the same reasons Gael is using ctypes - it's trivial. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Read-only mercurial mirror of numpy trunk available
> > Better solution is to keep the tags in a Mercurial Queues patch, as I did. > Good. As I said, I know bzr much better than hg, and I did the mirror to > get something started (and get used to hg, too). Since you know hg, it > is better than you maintain this for a while for people to try it out. I also just found that if I serve the hg repository generated by "hgpullsvn" directly, the tags are visible on the web, but not when you clone. I sometimes do hg mirrors of some svn projects just to browse the history conveniently, so this is handy. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Read-only mercurial mirror of numpy trunk available
On Jan 27, 2008 6:57 PM, Pauli Virtanen <[EMAIL PROTECTED]> wrote: > > la, 2008-01-19 kello 14:15 +0900, David Cournapeau kirjoitti: > > Pauli Virtanen wrote: > > > pe, 2008-01-18 kello 18:06 +0900, David Cournapeau kirjoitti: > > >> Hi there, > > >> > > >> I got a mercurial mirror of numpy available. I put some basic info > > >> on the wiki > > >> > > >> http://scipy.org/scipy/numpy/wiki/HgMirror > > > > > > Nice! > > > > Don't hesitate to put more info there. Although I would consider myself > > relatively proficient with bzr, mercurial has some few important > > differences that I am still not sure to understand (in particular the > > lack of "one branch is one directory" concept) > > I think I don't have edit access to the Trac Wiki... > > Anyway, it'd be useful to also have either hg or bzr read-only mirror of > scipy too! > > I'd actually like to see these mirrors hosted at some single "official" > place (svn.scipy.org?): If everyone maintains their own SVN mirror, the > resulting repositories typically turn out "unrelated" as far as the SCM > programs are concerned (because of different start points, different > conversion tools etc.), and one cannot easily pull and merge changes > from other people's repositories. > > > > hgsvn makes only repository-local svn tags. They go in .hg/localtags, > > > from where they can be copied manually. I don't think hgsvn has an > > > option to put the tags into .hgtags (from where they would be passed on > > > automatically when the repository is cloned). > > > > Since tags seem to be handled through plain files in mercurial, would > > copying localtags into .hgtags work ? > > It will work by just pasting the contents of localtags to .hgtags and > committing. However, this would need to be done periodically and would > pollute the history a bit. Better solution is to keep the tags in a Mercurial Queues patch, as I did. See for yourself: http://hg.certik.cz/numpy-hg/ BTW this mirror is much faster for me. You can browse the full history with tags online, you can clone like this: hg clone http://hg.certik.cz/numpy-hg/ It will convert the MQ patch to a regular commit, so you won't see the difference. But I, on the server, will just do: $ hg qpop Patch queue now empty $ hg pull pulling from /home/ondra/scipy/trunk searching for changes no changes found $ hg qpush applying tags.patch Now at: tags.patch Then I'll copy the updated localtags to .hgtags and do: $ hg qrefresh and that's it. No history pollution. If I decide I don't wont't the tags, I just do "hg qpop" and tags are gone. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to work with mercurial and numpy right now
Hi, if you want to play with Mercurial now (without forcing everyone else to leave svn), I suggest this: http://cheeseshop.python.org/pypi/hgsvn I tried that and it works. It's a very easy way to create a hg mirror at your computer. And then you can take this as the official upstream repository (which you don't have write access to). Whenever somone commits to the svn, you just do hgpullsvn and it updates your mercurial repo. Then you just clone it and create branches, for example the scons branch can be easily managed like this. Then you prepare patches, against your "official local mercurial mirror", using for example "hg export", or something, those patches should be possible to apply against the svn repository as well. You sent them for review and then (you or someone else) commit them using svn, then you'll "hgpullsvn" your local mercurial mirror and merge the changes to all your other branches. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Jan 6, 2008 12:55 AM, Bill Baxter <[EMAIL PROTECTED]> wrote: > On Jan 6, 2008 8:25 AM, Stefan van der Walt <[EMAIL PROTECTED]> wrote: > > I recall something you said to David last week, regarding merges with > > SVN: that a person never knows how to do it until *after* you've done > > it! We often make branches in scipy and numpy, and stand a lot to > > gain from a distributed RCS. > > > > Once a person knows how to use SVN, it doesn't take much effort at all > > to learn bzr or hg (even the commands are often the same). The main > > change is a mind-shift: that branches are now a lot friendlier, and > > that they are accessable to everybody. > > I understand that DVCS's do merging better. But what I don't really > understand is why this is an inherent advantage of DVCS. Isnt it > just a matter of the current crop of DVCS's implementing a better > merge algorithm than SVN? The SVN guys seem to be competent, so if > you just give them time surely they will eventually incorporate these > better merging algorithms into SVN. Who wouldn't want better merging? It's not just about merging. But anyway, all arguments were already said in this thread. I fully agree with both Davids and Fernando. So let's setup an official mercurial mirror, that will automatically download all svn commits. That way, we can easily work with Mercurial, clone the repos, browse history, everything. Review patches. And then, when the patches are reviewed, instead of pushing them to the Mercurial repo, they will be committed using svn. No big deal, everyone is happy. We did it too in sympy, at the beginning, because we were afraid of switching (it was mainly me, who was afraid, because I am very conservative, that's why I use Debian:). But then, once you try DVCS, you never want to come back. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Jan 5, 2008 8:15 PM, Fernando Perez <[EMAIL PROTECTED]> wrote: > On Jan 5, 2008 12:08 PM, David M. Cooke <[EMAIL PROTECTED]> wrote: > > On Jan 4, 2008, at 13:58 , Fernando Perez wrote: > > > > > My vote so far is for hg, for performance reasons but also partly > > > because sage and sympy already use it, two projects I'm likely to > > > interact a lot with and that are squarely in line with the > > > ipython/numpy/scipy/matplotlib world. Since they went first and made > > > the choice, I'm happy to let that be a factor in my decision. I'd > > > rather use a tool that others in the same community are also using, > > > especially when the choice is a sound one on technical merit alone. > > > > > > Just my 1e-2... > > > > > > +1 on mercurial. It's what I use these days (previously, I used darcs, > > which I still like for its patch-handling semantics, but its > > dependence on Haskell, and the dreaded exponential-time merge are a > > bit of a pain). > > Regarding the 'record' capapbilities of darcs which were indeed very > nice, here's something that was recently mentioned on the sage list: > > """ > I noticed that Mercurial 0.9.5 has a "record" extension that mimics the > darcs record functionality of interactively asking what changes you want > to commit out of a file. I know there was discussion of this a while ago. > > Reference: > > http://www.selenic.com/pipermail/mercurial/2007-October/015150.html > under the New extensions heading. See also > http://www.selenic.com/mercurial/wiki/index.cgi/RecordExtension > > Anyways, I'm just posting this as an FYI. It might be nice to expose > this functionality to sage, if we haven't already. > > Thanks, > > Jason > """ Kirill (a sympy developer) has also sent patches for qrecord (record for mercurial queues) http://www.selenic.com/pipermail/mercurial-devel/2007-December/003953.html Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Jan 4, 2008 10:05 PM, David Cournapeau <[EMAIL PROTECTED]> wrote: > On Jan 5, 2008 5:36 AM, Charles R Harris <[EMAIL PROTECTED]> wrote: > > > > > > > > A quick google for benchmarks show that a year ago, hg was a bit faster and > > generated smaller repositories than bzr, but I don't think the difference is > > enough to matter. > > Forget a year ago, because as far as bzr is concerned, they got much > faster (several times faster for common operations like > commit/branch/log/merge). > > > but Linus was definitely focused on speed, which is easy to understand if > > you look at the churn in the kernel. Anyway, I suspect that, technically, > > both bzr and hg are suitable choices. I'm not sure esr correct that it is > > unlikely that both are going to last long term, bazaar (the ancestor of bzr) > > is used for Ubuntu. But the two are similar and fill the same niche, so I > > expect that one or the other will become dominant in the wild. And hg seems > > to have the advantage of a head start and not being as tightly tied to > > Linux. > > bzr is not tied to linux. They always have win32 binaries, TortoiseBzr > has a longer history than the mercurial one, and as said previously, > one developer of bzr at least is mainly a windows user. I don't want > to sound like I defend bzr, because honestly, I don't care about which > one is used, but so far, the arguments I heard against bzr do not > reflect my experience at all. > > One thing that bzr tries hard is the general UI, and the explicit > support for several workflows (with moderately advanced concepts such > as shared repositories, bound branches: for example, with a branch A > bound to branch B, a commit is first pushed on branch B, and if > successfull, applied to A; for centralized worflows, this makes things > easier). I honestly do not know if this is significant. bzr claims its > merge capability is better: I do not know if this is true, or if that > matters at all. > > I would rather discuss those than "bzr is tied to linux", because I > don't think they are based on accurate or recent informations. As I > said, I have bzr imports of scipy and scikits, and I could easily to > the same for hg, make them available for everybody to play with. Instead of devising our own arguments, read this: http://bazaar-vcs.org/BzrVsHg and the mercurial response therein. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Jan 4, 2008 5:56 PM, David Cournapeau <[EMAIL PROTECTED]> wrote: > On Jan 5, 2008 1:30 AM, Charles R Harris <[EMAIL PROTECTED]> wrote: > > > > > I like Mercurial and use it a lot, but I'm not convinced we have enough > > developers and code to justify the pain of changing the VCS at this time. > > I don't understand the number of developers argument: on most of the > projects I am working on, I am the only developer, and I much prefer > bzr to svn, although for reasons which are not really relevant to a > project like numpy/scipy. > > > SVN g!enerally works well and has good support on Windows through tortoise. > That's where I don't agree: I don't think svn works really well. As > long as you use it as an history backup, it works ok, but that's it. > The non functional merge makes branching almost useless, reverting > back in time is extremely cumbersome, > > Mercurial also has tortoise support these days, but I haven't ever used it > > and can't comment on it. In fact, I've never even used Mercurial on windows, > > perhaps someone can relate their experiences with it. I suppose a shift > > might be justified if there is a lot of branch merging and such in our > > future. Anyone know what folks are working in branches? > Well, I started this discussion because of the scikits discussion. A > typical use of branches is for sandboxes: it makes a lot of sense to > use branches instead of sandboxes. Also, when branching actually > works, you really start using many branches: I do it all the time on > all my projects, and I am the only developer on most of them. It means > that you commit smaller changes (because comitting does not mean > makeing your changes available to the trunk), and instead of > submitting one big changeset, you actually submit a serie of small > changes. This really makes a big difference IMHO. Also, things like > log, blame are actually usable, since they are much faster on DVCS. > > For something like scipy (less for numpy), where many people develop > different things, I think it really makes a lot of sense to use a > DVCS. I actually think scipy to be more distributed in nature than > many open source projects (again, this is much less true for numpy, > IMHO). David is 100% right, I fully support this. I would be just repeating what he says. Charles actually said another point in favor of Mercurial - it works on Windows (at least people say so), while git not that much (at least people say so). I never use Windows myself, so I don't know. Subversion sucks not only in the merge thing, but especially when providing patches. Because most of the people don't have access to the repo, and not being able to commit locally (=work incrementally), is just bad. So, I use mercurial even when providing patches for svn. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
> Imagine the pain in the other direction, which was my experience :) I > actually did not believe at first that it was so bad, and thought I was > doing something wrong. At least, it certainly convinced me that SVN was > not easier than DVCS. It would made me sick. :) > I am not familiar with sympy: you are not using trac at all ? Also, how We use googlecode: http://code.google.com/p/sympy/ it works nice for us. > did you convert the svn history ? using the mercurial extension. Kirill submitted some patches, so that also branches are converted and tags too. > I like the mercurial's way of showing branches and co; bzr does not have > anything like that out of the box (there are separate projects to show > sources; there is also launchpad of course, but since it is not open > source, I do not even consider it for numpy/scipy). > > On the other hand, the bzr community is more user-friendly: the tool is > easier to use I think, the graphical tools are more advanced, at least > from my experience. I never used bzr, so I cannot judge. > > We were basically only deciding between git and Mercurial, but we > > chose mercurial, because > > > > * we are python guys and Mercurial is in python+C, very nicely written > > and they accept patches (Kirill, one sympy developer, has posted > > several already, to implement features he was missing - he used to use > > darcs before) > > * Sage uses it > For some time, the big problem of bzr was speed. But bzr accomplished > quite a lot the last year: the first time I used mercurial, the speed > difference was obvious; it is not so true anymore (they 'feel' the same, > basically, but I have not used mercurial extensively, at least compared > to bzr). > > So I think it really boils down to the difficulty of the transition, the > availability of third party tools (and also the tool used by other > projects similar to numpy, as you mentionned). I know that bzr is used by Canonical, but I think socially, you should choose either mercurial or git. Those are imho the most widespread DVCS. As I said, Mercurial being Python+C was very important for us, so that we can easily fix bugs and implement new functionality in mercurial. Also the commands of mercurial are very similar to svn. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
