Re: [Numpy-discussion] PSF GSoC 2010 (Py3K focus)
On 03/08/2010 11:39 PM, Charles R Harris wrote: On Mon, Mar 8, 2010 at 10:29 PM, Jarrod Millman mill...@berkeley.edu mailto:mill...@berkeley.edu wrote: I added Titus' email regarding the PSF's focus on Py3K-related projects to our SoC ideas wiki page: http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas Given Titus' email, this is the most likely list of projects we will get accepted this year: - finish porting NumPy to Py3K I think Numpy is pretty much done. It needs use testing to wring out any small oddities, but it doesn't look to me like a GSOC project at the moment. Maybe Pauli can weigh in here. - port SciPy to Py3K This project might be more appropriate, although I'm not clear on what needs to be done. - port matplotlib to Py3K - port ipython to Py3K Chuck +1 to all of these as these are probably next major roadblocks after numpy for Py3K usage. From the web page, having both f2py and/or Fwrap working with Py3K should be a specific goal (not that I use these). Also having Fwrap (as I understood the recent discussion) supporting the Fortran 77 used in Scipy. Bruce ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PSF GSoC 2010 (Py3K focus)
On Mon, Mar 8, 2010 at 11:39 PM, Charles R Harris charlesr.har...@gmail.com wrote: - port matplotlib to Py3K We'd be happy to mentor a project here. To my knowledge, nothing has been done, other than upgrade to CXX6 (our C++ extension lib). Most, but not all, of our extension code is exposed through CXX, which as of v6 is python3 compliant so that should help. But I suspect there is enough work to justify a GSOC project on the mpl side. JDH ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] printing structured arrays
Hello, thanks to all who responded and have their input here. I added a little code snippet to show the view and reshape: http://www.scipy.org/Cookbook/Recarray What do you think? Is this worth to go into the official docs? The page http://docs.scipy.org/doc/numpy/user/basics.rec.html is quite sparse... I still wonder why there is not a quick function for such a view / reshape conversion. Best regards, Timmie ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PSF GSoC 2010 (Py3K focus)
Mon, 08 Mar 2010 22:39:00 -0700, Charles R Harris wrote: On Mon, Mar 8, 2010 at 10:29 PM, Jarrod Millman mill...@berkeley.eduwrote: I added Titus' email regarding the PSF's focus on Py3K-related projects to our SoC ideas wiki page: http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas Given Titus' email, this is the most likely list of projects we will get accepted this year: - finish porting NumPy to Py3K I think Numpy is pretty much done. It needs use testing to wring out any small oddities, but it doesn't look to me like a GSOC project at the moment. Maybe Pauli can weigh in here. I think it's pretty much done. Even f2py should work. What's left is mostly testing it, and fixing any issues that crop up. Note that the SVN Numpy should preferably still see more testing on Python 2 against real-world packages that use it, before release. There's been a significant amount of code churn. - port SciPy to Py3K This project might be more appropriate, although I'm not clear on what needs to be done. I think porting Scipy proceeds in these steps: 1. Set up a similar 2to3-using build system for Python 3 as currently in Numpy. 2. Change the code so that it works both on Python 2 and Python 3. This can be done one submodule at a time. For C code, the changes required are mainly PyString usage. Some macros need to be defined to allow the same code to build on Py2 and Py3. Scipy is probably easier to port than Numpy here, since IIRC it doesn't mess much with strings, and its use of the Python C-API is much more limited. Also, parts written with Cython need essentially no porting. For Python code, the main issue is again bytes vs. unicode fixes, mainly inserting numpy.compat.asbytes() at correct locations to make e.g. bytes literals. All I/O code should be changed to work solely with Bytes. Since 2to3 takes care of most the other things, the remaining work is in fixing things it mishandles. I don't have a good estimate for how much work is in making Scipy work. I'd guess the work needed is probably slightly less than for Numpy. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Memory leak with matrices?
Hi, I've got two issues: First, the following seems to cause a memory leak, using numpy 1.3.0: a = matrix(ones(1)) while True: a += 0 This only seems to happen when a is a matrix rather than an array, and when the short hand '+=' is used. Second, I'm not sure whether that's a bug or whether I just don't understand what's going on, but when a is a column array, (e.g. a = ones((10, 1))), then a -= a[0,:] only subtracts from a[0, 0], whereas not using the short hand or using something else than a on the righthand side seems to subtract from all rows as expected. Thanks a lot, David -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] printing structured arrays
On Mon, Mar 8, 2010 at 5:50 PM, Tim Michelsen
timmichel...@gmx-topmail.de wrote:
Hello,
thanks to all who responded and have their input here.
I added a little code snippet to show the view and reshape:
http://www.scipy.org/Cookbook/Recarray
What do you think?
Is this worth to go into the official docs?
The page http://docs.scipy.org/doc/numpy/user/basics.rec.html is quite
sparse...
I still wonder why there is not a quick function for such a view /
reshape conversion.
Thanks, the docs for working with arrays with structured dtypes are sparse
Note that in your example .view(np.ndarray) doesn't do anything
struct_diffdtype[['str_var', 'x',
'y']].view(np.ndarray).reshape(len(struct_diffdtype),-1)
array([[(1.0, 'string1', 2.0)],
[(3.0, 'string2', 4.0996)]],
dtype=[('x', 'f8'), ('str_var', '|S7'), ('y', 'f8')])
struct_diffdtype[['x',
'y']].view(np.ndarray).reshape(len(struct_diffdtype),-1)
array([[(1.0, 2.0)],
[(3.0, 4.0996)]],
dtype=[('x', 'f8'), ('y', 'f8')])
view on columns with floating values (is this a copy???)
struct_diffdtype[['x', 'y']].view(float).reshape(len(struct_diffdtype),-1)
array([[ 1. , 2. ],
[ 3. , 4.1]])
and float view on strings is not possible
struct_diffdtype[['str_var', 'x',
'y']].view(float).reshape(len(struct_diffdtype),-1)
Traceback (most recent call last):
ValueError: new type not compatible with array.
Josef
Best regards,
Timmie
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak with matrices?
On Tue, Mar 9, 2010 at 12:31 PM, David Paul Reichert d.p.reich...@sms.ed.ac.uk wrote: Hi, I've got two issues: First, the following seems to cause a memory leak, using numpy 1.3.0: a = matrix(ones(1)) while True: a += 0 This only seems to happen when a is a matrix rather than an array, and when the short hand '+=' is used. Second, I'm not sure whether that's a bug or whether I just don't understand what's going on, but when a is a column array, (e.g. a = ones((10, 1))), then a -= a[0,:] only subtracts from a[0, 0], whereas not using the short hand or using something else than a on the righthand side seems to subtract from all rows as expected. this is because a[0,0] is set to zero after the first inplace subtraction, then zero is subtracted from all other rows a = np.ones((10, 1)) a array([[ 1.], [ 1.], [ 1.], [ 1.], [ 1.], [ 1.], [ 1.], [ 1.], [ 1.], [ 1.]]) a += a[0,:] a array([[ 2.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.]]) a -= a[0,:] a array([[ 0.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.]]) Josef Thanks a lot, David -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak with matrices?
Thanks for the reply. Yes never mind the second issue, I had myself confused there. Any comments on the memory leak? On Tue, Mar 9, 2010 at 5:55 PM, josef.p...@gmail.com wrote: On Tue, Mar 9, 2010 at 12:31 PM, David Paul Reichert d.p.reich...@sms.ed.ac.uk wrote: Hi, I've got two issues: First, the following seems to cause a memory leak, using numpy 1.3.0: a = matrix(ones(1)) while True: a += 0 This only seems to happen when a is a matrix rather than an array, and when the short hand '+=' is used. Second, I'm not sure whether that's a bug or whether I just don't understand what's going on, but when a is a column array, (e.g. a = ones((10, 1))), then a -= a[0,:] only subtracts from a[0, 0], whereas not using the short hand or using something else than a on the righthand side seems to subtract from all rows as expected. this is because a[0,0] is set to zero after the first inplace subtraction, then zero is subtracted from all other rows a = np.ones((10, 1)) a array([[ 1.], [ 1.], [ 1.], [ 1.], [ 1.], [ 1.], [ 1.], [ 1.], [ 1.], [ 1.]]) a += a[0,:] a array([[ 2.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.]]) a -= a[0,:] a array([[ 0.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.], [ 3.]]) Josef Thanks a lot, David -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PSF GSoC 2010 (Py3K focus)
On Tue, Mar 9, 2010 at 9:02 AM, Pauli Virtanen pav...@iki.fi wrote: Mon, 08 Mar 2010 22:39:00 -0700, Charles R Harris wrote: On Mon, Mar 8, 2010 at 10:29 PM, Jarrod Millman mill...@berkeley.eduwrote: I added Titus' email regarding the PSF's focus on Py3K-related projects to our SoC ideas wiki page: http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas Given Titus' email, this is the most likely list of projects we will get accepted this year: - finish porting NumPy to Py3K I think Numpy is pretty much done. It needs use testing to wring out any small oddities, but it doesn't look to me like a GSOC project at the moment. Maybe Pauli can weigh in here. I think it's pretty much done. Even f2py should work. What's left is mostly testing it, and fixing any issues that crop up. Note that the SVN Numpy should preferably still see more testing on Python 2 against real-world packages that use it, before release. There's been a significant amount of code churn. - port SciPy to Py3K This project might be more appropriate, although I'm not clear on what needs to be done. I think porting Scipy proceeds in these steps: 1. Set up a similar 2to3-using build system for Python 3 as currently in Numpy. 2. Change the code so that it works both on Python 2 and Python 3. This can be done one submodule at a time. For C code, the changes required are mainly PyString usage. Some macros need to be defined to allow the same code to build on Py2 and Py3. Scipy is probably easier to port than Numpy here, since IIRC it doesn't mess much with strings, and its use of the Python C-API is much more limited. Also, parts written with Cython need essentially no porting. For Python code, the main issue is again bytes vs. unicode fixes, mainly inserting numpy.compat.asbytes() at correct locations to make e.g. bytes literals. All I/O code should be changed to work solely with Bytes. Since 2to3 takes care of most the other things, the remaining work is in fixing things it mishandles. I don't have a good estimate for how much work is in making Scipy work. I'd guess the work needed is probably slightly less than for Numpy. a few questions: Is scipy.special difficult or time consuming to port? In the build system, is it possible not to build some subpackages that might be slow in being ported, e.g. ndimage, weave? Is there a good utility script to check dependencies between subpackages? (import scipy.stats loads a very large number of modules) import sys len(sys.modules) 125 import numpy len(sys.modules) 259 import scipy len(sys.modules) 339 import scipy.stats len(sys.modules) 556 Josef -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak with matrices?
On Tue, Mar 9, 2010 at 11:31, David Paul Reichert
d.p.reich...@sms.ed.ac.uk wrote:
Hi,
I've got two issues:
First, the following seems to cause a memory leak,
using numpy 1.3.0:
a = matrix(ones(1))
while True:
a += 0
This only seems to happen when a is a matrix rather
than an array, and when the short hand '+=' is used.
Yes, I can verify that there is a leak in the current SVN, too. It
appears that we are creating dictionaries with the value {'_getitem':
False} and never decrefing them. The matrix object itself owns the
reference.
Second, I'm not sure whether that's a bug or whether
I just don't understand what's going on, but when a is a column
array, (e.g. a = ones((10, 1))), then
a -= a[0,:]
only subtracts from a[0, 0], whereas not using the short hand
or using something else than a on the righthand side seems
to subtract from all rows as expected.
a[0,:] creates a view onto the same memory of the original array.
Since you modify the values in-place, a[0,0] gets set to
a[0,0]-a[0,0]==0, then a[1,0] gets set to a[1,0] - a[0,0] == a[1,0] -
0 == a[1,0], etc. Try this instead:
a -= a[0,:].copy()
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth.
-- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak with matrices?
On Tue, Mar 9, 2010 at 1:28 PM, Robert Kern robert.k...@gmail.com wrote:
On Tue, Mar 9, 2010 at 11:31, David Paul Reichert
d.p.reich...@sms.ed.ac.uk wrote:
Hi,
I've got two issues:
First, the following seems to cause a memory leak,
using numpy 1.3.0:
a = matrix(ones(1))
while True:
a += 0
This only seems to happen when a is a matrix rather
than an array, and when the short hand '+=' is used.
Yes, I can verify that there is a leak in the current SVN, too. It
appears that we are creating dictionaries with the value {'_getitem':
False} and never decrefing them. The matrix object itself owns the
reference.
With numpy 1.4.0 memory usage also keeps growing until I kill the process.
Josef
Second, I'm not sure whether that's a bug or whether
I just don't understand what's going on, but when a is a column
array, (e.g. a = ones((10, 1))), then
a -= a[0,:]
only subtracts from a[0, 0], whereas not using the short hand
or using something else than a on the righthand side seems
to subtract from all rows as expected.
a[0,:] creates a view onto the same memory of the original array.
Since you modify the values in-place, a[0,0] gets set to
a[0,0]-a[0,0]==0, then a[1,0] gets set to a[1,0] - a[0,0] == a[1,0] -
0 == a[1,0], etc. Try this instead:
a -= a[0,:].copy()
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth.
-- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] printing structured arrays
Tim Michelsen wrote: I still wonder why there is not a quick function for such a view / reshape conversion. Because it is difficult (impossible?) to do in the general case. .view() really isn't that bad, in fact, it remarkably powerful and flexible! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] crash at prompt exit after running test
hi there, I just installed the current head of numpy and built it. trying python and then import numpy, and then CTRL-D to exit, all goes well. But doing the same with a numpy.test() before CTRL-D ends up in : clip Ran 2892 tests in 35.814s OK (KNOWNFAIL=4, SKIP=6) nose.result.TextTestResult run=2892 errors=0 failures=0 python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Aborted (core dumped) Does that ring a bell to any of you? My system is Fedora 12, I used gfortran to build and In [10]: sys.path Out[10]: ['', '/home/cohen/.local/bin', '/home/cohen/sources/python/ipython', '/usr/lib/python26.zip', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/home/cohen/.local/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/PIL', '/usr/lib/python2.6/site-packages/gst-0.10', '/usr/lib/python2.6/site-packages/gtk-2.0', '/usr/lib/python2.6/site-packages/webkit-1.0', '/usr/lib/python2.6/site-packages/wx-2.8-gtk2-unicode', '/home/cohen/sources/python/ipython/IPython/extensions', u'/home/cohen/.ipython'] numpy and ipython live in $HOME/.local thanks in advance, Johann ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] crash at prompt exit after running test
On Tue, Mar 9, 2010 at 13:30, Johann Cohen-Tanugi co...@lpta.in2p3.fr wrote: hi there, I just installed the current head of numpy and built it. trying python and then import numpy, and then CTRL-D to exit, all goes well. But doing the same with a numpy.test() before CTRL-D ends up in : clip Ran 2892 tests in 35.814s OK (KNOWNFAIL=4, SKIP=6) nose.result.TextTestResult run=2892 errors=0 failures=0 python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Aborted (core dumped) Does that ring a bell to any of you? Nope! Can you show us a gdb backtrace of the crash? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] crash at prompt exit after running test
I have tried to localize the core dump in vain any idea where I
should look for it?
I did not manage to catch it with pdb :
[co...@jarrett ~]$ .local/bin/ipython
Python 2.6.2 (r262:71600, Jan 25 2010, 18:46:45)
Type copyright, credits or license for more information.
IPython 0.11.alpha1.bzr.r1223 -- An enhanced Interactive Python.
? - Introduction and overview of IPython's features.
%quickref - Quick reference.
help - Python's own help system.
object? - Details about 'object'. ?object also works, ?? prints more.
In [1]: import pdb
In [2]: pdb.set_trace()
--Call--
/home/cohen/sources/python/ipython/IPython/core/prompts.py(525)__call__()
- def __call__(self,arg=None):
(Pdb) import numpy
(Pdb) numpy.test()
clip
Ran 2892 tests in 35.888s
OK (KNOWNFAIL=4, SKIP=6)
nose.result.TextTestResult run=2892 errors=0 failures=0
(Pdb) exit()
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (137, 0))
---
BdbQuit Traceback (most recent call last)
/home/cohen/ipython console in module()
/home/cohen/sources/python/ipython/IPython/core/prompts.pyc in
__call__(self, arg)
523 self.prompt_out.set_colors()
524
-- 525 def __call__(self,arg=None):
526 Printing with history cache management.
527
/usr/lib/python2.6/bdb.pyc in trace_dispatch(self, frame, event, arg)
46 return self.dispatch_line(frame)
47 if event == 'call':
--- 48 return self.dispatch_call(frame, arg)
49 if event == 'return':
50 return self.dispatch_return(frame, arg)
/usr/lib/python2.6/bdb.pyc in dispatch_call(self, frame, arg)
76 return # None
77 self.user_call(frame, arg)
--- 78 if self.quitting: raise BdbQuit
79 return self.trace_dispatch
80
BdbQuit:
In [3]:
Do you really want to exit ([y]/n)?
python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs
!= 0' failed.
Aborted (core dumped)
Note that I have several warnings during test():
Warning: divide by zero encountered in power
Warning: divide by zero encountered in power
Warning: divide by zero encountered in power
...Warning:
invalid value encountered in sqrt
..Warning: invalid value encountered in sqrt
etc
I dont think it is related though
Johann
On 03/09/2010 08:35 PM, Robert Kern wrote:
On Tue, Mar 9, 2010 at 13:30, Johann Cohen-Tanugico...@lpta.in2p3.fr wrote:
hi there,
I just installed the current head of numpy and built it. trying
python and then import numpy, and then CTRL-D to exit, all goes well.
But doing the same with a numpy.test() before CTRL-D ends up in :
clip
Ran 2892 tests in 35.814s
OK (KNOWNFAIL=4, SKIP=6)
nose.result.TextTestResult run=2892 errors=0 failures=0
python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs
!= 0' failed.
Aborted (core dumped)
Does that ring a bell to any of you?
Nope! Can you show us a gdb backtrace of the crash?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] crash at prompt exit after running test
On Tue, Mar 9, 2010 at 13:50, Johann Cohen-Tanugi co...@lpta.in2p3.fr wrote: I have tried to localize the core dump in vain any idea where I should look for it? I did not manage to catch it with pdb : Not pdb, gdb. $ gdb python ... (gdb) run Starting program ... ... # Possibly another (gdb) prompt: (gdb) continue # - Type this. Python 2.6.2 ... import numpy # - Type this and do whatever else is necessary to reproduce the crash. ... (gdb) bt # - Type this. # - Copy-paste these results here. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in signal.convolve2d? Alternative?
On Tue, Mar 9, 2010 at 13:49, David Reichert d.p.reich...@sms.ed.ac.uk wrote: Hi, I just reported a memory leak with matrices, and I might have found another (unrelated) one in the convolve2d function: import scipy.signal from numpy import ones while True: scipy.signal.convolve2d(ones((1,1)), ones((1,1))) This does not leak for me using SVN versions of numpy and scipy. I recommend upgrading. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] crash at prompt exit after running test
thanks Robert, here it is : exit() python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Program received signal SIGABRT, Aborted. 0x004a1416 in __kernel_vsyscall () Missing separate debuginfos, use: debuginfo-install atlas-3.8.3-12.fc12.i686 compat-libf2c-34-3.4.6-18.i686 keyutils-libs-1.2-6.fc12.i686 krb5-libs-1.7.1-2.fc12.i686 libcom_err-1.41.9-7.fc12.i686 libgcc-4.4.3-4.fc12.i686 libgfortran-4.4.3-4.fc12.i686 libselinux-2.0.90-5.fc12.i686 (gdb) bt #0 0x004a1416 in __kernel_vsyscall () #1 0x00609a91 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #2 0x0060b35a in abort () at abort.c:92 #3 0x00602be8 in __assert_fail (assertion=value optimized out, file=value optimized out, line=value optimized out, function=value optimized out) at assert.c:81 #4 0x0032e31e in visit_decref (op=value optimized out, data=value optimized out) at Modules/gcmodule.c:277 #5 0x002a18c2 in dict_traverse (op=value optimized out, visit=value optimized out, arg=value optimized out) at Objects/dictobject.c:2003 #6 0x0032eaf3 in subtract_refs (generation=value optimized out) at Modules/gcmodule.c:296 #7 collect (generation=value optimized out) at Modules/gcmodule.c:817 #8 0x0032f640 in PyGC_Collect () at Modules/gcmodule.c:1292 #9 0x003200f0 in Py_Finalize () at Python/pythonrun.c:424 #10 0x00320218 in Py_Exit (sts=value optimized out) at Python/pythonrun.c:1714 #11 0x00320367 in handle_system_exit () at Python/pythonrun.c:1116 #12 0x0032057d in PyErr_PrintEx (set_sys_last_vars=value optimized out) at Python/pythonrun.c:1126 #13 0x0032079f in PyErr_Print () at Python/pythonrun.c:1035 #14 0x00320f9a in PyRun_InteractiveOneFlags (fp=value optimized out, filename=value optimized out, flags=value optimized out) at Python/pythonrun.c:843 #15 0x00321093 in PyRun_InteractiveLoopFlags (fp=value optimized out, filename=value optimized out, flags=0xbfffefec) at Python/pythonrun.c:760 #16 0x003211df in PyRun_AnyFileExFlags (fp=value optimized out, filename=0x365add stdin, closeit=value optimized out, flags=value optimized out) at Python/pythonrun.c:729 #17 0x0032dc85 in Py_Main (argc=value optimized out, argv=value optimized out) at Modules/main.c:599 #18 0x080485c8 in main (argc=value optimized out, argv=value optimized out) at Modules/python.c:23 I also saw things like: warning: .dynamic section for /lib/libcom_err.so.2 is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for /lib/libkeyutils.so.1 is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations Johann On 03/09/2010 08:56 PM, Robert Kern wrote: On Tue, Mar 9, 2010 at 13:50, Johann Cohen-Tanugico...@lpta.in2p3.fr wrote: I have tried to localize the core dump in vain any idea where I should look for it? I did not manage to catch it with pdb : Not pdb, gdb. $ gdb python ... (gdb) run Starting program ... ... # Possibly another (gdb) prompt: (gdb) continue #- Type this. Python 2.6.2 ... import numpy #- Type this and do whatever else is necessary to reproduce the crash. ... (gdb) bt #- Type this. #- Copy-paste these results here. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] crash at prompt exit after running test
thinking about it, this morning there was a fedora update to python, so I am using 2.6.2-4.fc12. Looks like the problem is in python itself, hence this piece of info. HTH, Johann On 03/09/2010 09:07 PM, Johann Cohen-Tanugi wrote: thanks Robert, here it is : exit() python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Program received signal SIGABRT, Aborted. 0x004a1416 in __kernel_vsyscall () Missing separate debuginfos, use: debuginfo-install atlas-3.8.3-12.fc12.i686 compat-libf2c-34-3.4.6-18.i686 keyutils-libs-1.2-6.fc12.i686 krb5-libs-1.7.1-2.fc12.i686 libcom_err-1.41.9-7.fc12.i686 libgcc-4.4.3-4.fc12.i686 libgfortran-4.4.3-4.fc12.i686 libselinux-2.0.90-5.fc12.i686 (gdb) bt #0 0x004a1416 in __kernel_vsyscall () #1 0x00609a91 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #2 0x0060b35a in abort () at abort.c:92 #3 0x00602be8 in __assert_fail (assertion=value optimized out, file=value optimized out, line=value optimized out, function=value optimized out) at assert.c:81 #4 0x0032e31e in visit_decref (op=value optimized out, data=value optimized out) at Modules/gcmodule.c:277 #5 0x002a18c2 in dict_traverse (op=value optimized out, visit=value optimized out, arg=value optimized out) at Objects/dictobject.c:2003 #6 0x0032eaf3 in subtract_refs (generation=value optimized out) at Modules/gcmodule.c:296 #7 collect (generation=value optimized out) at Modules/gcmodule.c:817 #8 0x0032f640 in PyGC_Collect () at Modules/gcmodule.c:1292 #9 0x003200f0 in Py_Finalize () at Python/pythonrun.c:424 #10 0x00320218 in Py_Exit (sts=value optimized out) at Python/pythonrun.c:1714 #11 0x00320367 in handle_system_exit () at Python/pythonrun.c:1116 #12 0x0032057d in PyErr_PrintEx (set_sys_last_vars=value optimized out) at Python/pythonrun.c:1126 #13 0x0032079f in PyErr_Print () at Python/pythonrun.c:1035 #14 0x00320f9a in PyRun_InteractiveOneFlags (fp=value optimized out, filename=value optimized out, flags=value optimized out) at Python/pythonrun.c:843 #15 0x00321093 in PyRun_InteractiveLoopFlags (fp=value optimized out, filename=value optimized out, flags=0xbfffefec) at Python/pythonrun.c:760 #16 0x003211df in PyRun_AnyFileExFlags (fp=value optimized out, filename=0x365add stdin, closeit=value optimized out, flags=value optimized out) at Python/pythonrun.c:729 #17 0x0032dc85 in Py_Main (argc=value optimized out, argv=value optimized out) at Modules/main.c:599 #18 0x080485c8 in main (argc=value optimized out, argv=value optimized out) at Modules/python.c:23 I also saw things like: warning: .dynamic section for /lib/libcom_err.so.2 is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for /lib/libkeyutils.so.1 is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations Johann On 03/09/2010 08:56 PM, Robert Kern wrote: On Tue, Mar 9, 2010 at 13:50, Johann Cohen-Tanugico...@lpta.in2p3.fr wrote: I have tried to localize the core dump in vain any idea where I should look for it? I did not manage to catch it with pdb : Not pdb, gdb. $ gdb python ... (gdb) run Starting program ... ... # Possibly another (gdb) prompt: (gdb) continue #- Type this. Python 2.6.2 ... import numpy #- Type this and do whatever else is necessary to reproduce the crash. ... (gdb) bt #- Type this. #- Copy-paste these results here. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] printing structured arrays
I still wonder why there is not a quick function for such a view / reshape conversion. Because it is difficult (impossible?) to do in the general case. .view() really isn't that bad, in fact, it remarkably powerful and flexible! I would not drop .view() but rather add a convenience function for struct_1dtype_float_alt = struct_1dtype.view((np.float, len(struct_1dtype.dtype.names))) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in signal.convolve2d? Alternative?
Hm, upgrading scipy from 0.7.0 to 0.7.1 didn't do the trick for me (still running numpy 1.3.0). I'm not sure if I feel confident enough to use developer versions, but I'll look into it. Cheers David On Tue, Mar 9, 2010 at 7:57 PM, Robert Kern robert.k...@gmail.com wrote: On Tue, Mar 9, 2010 at 13:49, David Reichert d.p.reich...@sms.ed.ac.uk wrote: Hi, I just reported a memory leak with matrices, and I might have found another (unrelated) one in the convolve2d function: import scipy.signal from numpy import ones while True: scipy.signal.convolve2d(ones((1,1)), ones((1,1))) This does not leak for me using SVN versions of numpy and scipy. I recommend upgrading. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in signal.convolve2d? Alternative?
On Tue, Mar 9, 2010 at 4:24 PM, David Reichert d.p.reich...@sms.ed.ac.uk wrote: Hm, upgrading scipy from 0.7.0 to 0.7.1 didn't do the trick for me (still running numpy 1.3.0). I'm not sure if I feel confident enough to use developer versions, but I'll look into it. If you don't need the extra options, you could also use in the meantime the nd version signal.convolve or the fft version signal.fftconvolve Josef Cheers David On Tue, Mar 9, 2010 at 7:57 PM, Robert Kern robert.k...@gmail.com wrote: On Tue, Mar 9, 2010 at 13:49, David Reichert d.p.reich...@sms.ed.ac.uk wrote: Hi, I just reported a memory leak with matrices, and I might have found another (unrelated) one in the convolve2d function: import scipy.signal from numpy import ones while True: scipy.signal.convolve2d(ones((1,1)), ones((1,1))) This does not leak for me using SVN versions of numpy and scipy. I recommend upgrading. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] printing structured arrays
josef.p...@gmail.com schrieb: On Mon, Mar 8, 2010 at 5:50 PM, Tim Michelsen timmichel...@gmx-topmail.de wrote: Hello, thanks to all who responded and have their input here. I added a little code snippet to show the view and reshape: http://www.scipy.org/Cookbook/Recarray What do you think? Is this worth to go into the official docs? The page http://docs.scipy.org/doc/numpy/user/basics.rec.html is quite sparse... I still wonder why there is not a quick function for such a view / reshape conversion. Thanks, the docs for working with arrays with structured dtypes are sparse Note that in your example .view(np.ndarray) doesn't do anything Please note that I wanted to demonstrate many different ways of view reshape. I updated the page: http://www.scipy.org/Cookbook/Recarray ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] printing structured arrays
Is this worth to go into the official docs? The page http://docs.scipy.org/doc/numpy/user/basics.rec.html is quite sparse... I still wonder why there is not a quick function for such a view / reshape conversion. Thanks, the docs for working with arrays with structured dtypes are sparse I cannot recover my longin for docs.scipy.org. Would you advice to add the elaborated example? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] crash at prompt exit after running test
ti, 2010-03-09 kello 21:14 +0100, Johann Cohen-Tanugi kirjoitti:
thinking about it, this morning there was a fedora update to python, so
I am using 2.6.2-4.fc12.
Looks like the problem is in python itself, hence this piece of info.
That the problem would be in Python is not so clear to me. Can you try
running it with the previous Python shipped by Fedora? Do you see the
problem then? What's the previous version, btw?
Memory errors are somewhat difficult to debug. Can you try running only
a certain subset of the tests, first
nosetests numpy.core
Be sure to set Pythonpath so that you get the correct Numpy version. If
it segfaults, proceed to (under numpy/core/tests)
nosetests test_multiarray.py
nosetests test_multiarray.py:TestNewBufferProtocol
Since the crash occurs in cyclic garbage collection, I'm doubting a bit
the numpy/core/src/multiarray/numpymemoryview.c implementation, since
that's the only part in Numpy that supports that.
Alternatively, just replace numpymemoryview.c with the attached one
which has cyclic GC stripped, and see if you still get the crash.
Cheers,
Pauli
/*
* Simple PyMemoryView'ish object for Python 2.6 compatibility.
*
* On Python = 2.7, we can use the actual PyMemoryView objects.
*
* Some code copied from the CPython implementation.
*/
#define PY_SSIZE_T_CLEAN
#include Python.h
#include structmember.h
#define _MULTIARRAYMODULE
#define NPY_NO_PREFIX
#include numpy/arrayobject.h
#include numpy/arrayscalars.h
#include npy_config.h
#include npy_3kcompat.h
#include numpymemoryview.h
#if (PY_VERSION_HEX = 0x0206) (PY_VERSION_HEX 0x0207)
/*
* Memory allocation
*/
static void
memorysimpleview_dealloc(PyMemorySimpleViewObject *self)
{
Py_CLEAR(self-base);
if (self-view.obj != NULL) {
PyBuffer_Release(self-view);
self-view.obj = NULL;
}
}
static PyObject *
memorysimpleview_new(PyTypeObject *subtype, PyObject *args, PyObject *kwds)
{
PyObject *obj;
static char *kwlist[] = {object, 0};
if (!PyArg_ParseTupleAndKeywords(args, kwds, O:memorysimpleview, kwlist,
obj)) {
return NULL;
}
return PyMemorySimpleView_FromObject(obj);
}
/*
* Buffer interface
*/
static int
memorysimpleview_getbuffer(PyMemorySimpleViewObject *self,
Py_buffer *view, int flags)
{
return PyObject_GetBuffer(self-base, view, flags);
}
static void
memorysimpleview_releasebuffer(PyMemorySimpleViewObject *self,
Py_buffer *view)
{
PyBuffer_Release(view);
}
static PyBufferProcs memorysimpleview_as_buffer = {
(readbufferproc)0, /*bf_getreadbuffer*/
(writebufferproc)0, /*bf_getwritebuffer*/
(segcountproc)0,/*bf_getsegcount*/
(charbufferproc)0, /*bf_getcharbuffer*/
(getbufferproc)memorysimpleview_getbuffer, /* bf_getbuffer */
(releasebufferproc)memorysimpleview_releasebuffer, /* bf_releasebuffer */
};
/*
* Getters
*/
static PyObject *
_IntTupleFromSsizet(int len, Py_ssize_t *vals)
{
int i;
PyObject *o;
PyObject *intTuple;
if (vals == NULL) {
Py_INCREF(Py_None);
return Py_None;
}
intTuple = PyTuple_New(len);
if (!intTuple) return NULL;
for(i=0; ilen; i++) {
o = PyInt_FromSsize_t(vals[i]);
if (!o) {
Py_DECREF(intTuple);
return NULL;
}
PyTuple_SET_ITEM(intTuple, i, o);
}
return intTuple;
}
static PyObject *
memorysimpleview_format_get(PyMemorySimpleViewObject *self)
{
return PyUString_FromString(self-view.format);
}
static PyObject *
memorysimpleview_itemsize_get(PyMemorySimpleViewObject *self)
{
return PyLong_FromSsize_t(self-view.itemsize);
}
static PyObject *
memorysimpleview_shape_get(PyMemorySimpleViewObject *self)
{
return _IntTupleFromSsizet(self-view.ndim, self-view.shape);
}
static PyObject *
memorysimpleview_strides_get(PyMemorySimpleViewObject *self)
{
return _IntTupleFromSsizet(self-view.ndim, self-view.strides);
}
static PyObject *
memorysimpleview_suboffsets_get(PyMemorySimpleViewObject *self)
{
return _IntTupleFromSsizet(self-view.ndim, self-view.suboffsets);
}
static PyObject *
memorysimpleview_readonly_get(PyMemorySimpleViewObject *self)
{
return PyBool_FromLong(self-view.readonly);
}
static PyObject *
memorysimpleview_ndim_get(PyMemorySimpleViewObject *self)
{
return PyLong_FromLong(self-view.ndim);
}
static PyGetSetDef memorysimpleview_getsets[] =
{
{format, (getter)memorysimpleview_format_get, NULL, NULL, NULL},
{itemsize, (getter)memorysimpleview_itemsize_get, NULL, NULL, NULL},
{shape, (getter)memorysimpleview_shape_get, NULL, NULL, NULL},
{strides, (getter)memorysimpleview_strides_get, NULL, NULL, NULL},
{suboffsets, (getter)memorysimpleview_suboffsets_get, NULL, NULL, NULL},
{readonly, (getter)memorysimpleview_readonly_get, NULL, NULL, NULL},
{ndim,
Re: [Numpy-discussion] crash at prompt exit after running test
On 03/10/2010 12:07 AM, Pauli Virtanen wrote: ti, 2010-03-09 kello 21:14 +0100, Johann Cohen-Tanugi kirjoitti: thinking about it, this morning there was a fedora update to python, so I am using 2.6.2-4.fc12. Looks like the problem is in python itself, hence this piece of info. That the problem would be in Python is not so clear to me. Can you try running it with the previous Python shipped by Fedora? Do you see the problem then? What's the previous version, btw? 2.6.2-1 IIRC. I would have to check, and I am not sure how to either query this information or step back one update up with yum :( Memory errors are somewhat difficult to debug. Can you try running only a certain subset of the tests, first nosetests numpy.core crash Be sure to set Pythonpath so that you get the correct Numpy version. If it segfaults, proceed to (under numpy/core/tests) nosetests test_multiarray.py nosetests test_multiarray.py:TestNewBufferProtocol neither crash, so the problem is not there Since the crash occurs in cyclic garbage collection, I'm doubting a bit the numpy/core/src/multiarray/numpymemoryview.c implementation, since that's the only part in Numpy that supports that. Alternatively, just replace numpymemoryview.c with the attached one which has cyclic GC stripped, and see if you still get the crash. Cheers, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] crash at prompt exit after running test
On 03/10/2010 12:33 AM, Johann Cohen-Tanugi wrote: On 03/10/2010 12:07 AM, Pauli Virtanen wrote: ti, 2010-03-09 kello 21:14 +0100, Johann Cohen-Tanugi kirjoitti: thinking about it, this morning there was a fedora update to python, so I am using 2.6.2-4.fc12. Looks like the problem is in python itself, hence this piece of info. That the problem would be in Python is not so clear to me. Can you try running it with the previous Python shipped by Fedora? Do you see the problem then? What's the previous version, btw? 2.6.2-1 IIRC. I would have to check, and I am not sure how to either query this information or step back one update up with yum :( Memory errors are somewhat difficult to debug. Can you try running only a certain subset of the tests, first nosetests numpy.core crash Be sure to set Pythonpath so that you get the correct Numpy version. If it segfaults, proceed to (under numpy/core/tests) nosetests test_multiarray.py nosetests test_multiarray.py:TestNewBufferProtocol neither crash, so the problem is not there I followed your lead and tried each script and ended up with : [co...@jarrett tests]$ nosetests test_ufunc.py . -- Ran 13 tests in 1.146s OK python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Aborted (core dumped) so test_ufunc.py seems to be at stake Since the crash occurs in cyclic garbage collection, I'm doubting a bit the numpy/core/src/multiarray/numpymemoryview.c implementation, since that's the only part in Numpy that supports that. Alternatively, just replace numpymemoryview.c with the attached one which has cyclic GC stripped, and see if you still get the crash. Cheers, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] crash at prompt exit after running test
more fun : [co...@jarrett tests]$ pwd /home/cohen/sources/python/numpy/numpy/core/tests [co...@jarrett tests]$ python -c 'import test_ufunc' python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Aborted (core dumped) and in the debugger: (gdb) run Starting program: /usr/bin/python warning: .dynamic section for /lib/libpthread.so.0 is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for /lib/libdl.so.2 is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for /lib/libc.so.6 is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations [Thread debugging using libthread_db enabled] Python 2.6.2 (r262:71600, Jan 25 2010, 18:46:45) [GCC 4.4.2 20091222 (Red Hat 4.4.2-20)] on linux2 Type help, copyright, credits or license for more information. import test_ufunc python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Program received signal SIGABRT, Aborted. 0x00aab416 in __kernel_vsyscall () Missing separate debuginfos, use: debuginfo-install atlas-3.8.3-12.fc12.i686 libgcc-4.4.3-4.fc12.i686 libgfortran-4.4.3-4.fc12.i686 (gdb) bt #0 0x00aab416 in __kernel_vsyscall () #1 0x00159a91 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #2 0x0015b35a in abort () at abort.c:92 #3 0x00152be8 in __assert_fail (assertion=value optimized out, file=value optimized out, line=value optimized out, function=value optimized out) at assert.c:81 #4 0x0050931e in visit_decref (op=value optimized out, data=value optimized out) at Modules/gcmodule.c:277 #5 0x0047c8c2 in dict_traverse (op=value optimized out, visit=value optimized out, arg=value optimized out) at Objects/dictobject.c:2003 #6 0x00509af3 in subtract_refs (generation=value optimized out) at Modules/gcmodule.c:296 #7 collect (generation=value optimized out) at Modules/gcmodule.c:817 #8 0x0050a640 in PyGC_Collect () at Modules/gcmodule.c:1292 #9 0x004fb0f0 in Py_Finalize () at Python/pythonrun.c:424 #10 0x0050868f in Py_Main (argc=value optimized out, argv=value optimized out) at Modules/main.c:625 #11 0x080485c8 in main (argc=value optimized out, argv=value optimized out) at Modules/python.c:23 which looks identical to the bt I sent to Robert earlier on. HTH, Johann On 03/10/2010 12:43 AM, Johann Cohen-Tanugi wrote: On 03/10/2010 12:33 AM, Johann Cohen-Tanugi wrote: On 03/10/2010 12:07 AM, Pauli Virtanen wrote: ti, 2010-03-09 kello 21:14 +0100, Johann Cohen-Tanugi kirjoitti: thinking about it, this morning there was a fedora update to python, so I am using 2.6.2-4.fc12. Looks like the problem is in python itself, hence this piece of info. That the problem would be in Python is not so clear to me. Can you try running it with the previous Python shipped by Fedora? Do you see the problem then? What's the previous version, btw? 2.6.2-1 IIRC. I would have to check, and I am not sure how to either query this information or step back one update up with yum :( Memory errors are somewhat difficult to debug. Can you try running only a certain subset of the tests, first nosetests numpy.core crash Be sure to set Pythonpath so that you get the correct Numpy version. If it segfaults, proceed to (under numpy/core/tests) nosetests test_multiarray.py nosetests test_multiarray.py:TestNewBufferProtocol neither crash, so the problem is not there I followed your lead and tried each script and ended up with : [co...@jarrett tests]$ nosetests test_ufunc.py . -- Ran 13 tests in 1.146s OK python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Aborted (core dumped) so test_ufunc.py seems to be at stake Since the crash occurs in cyclic garbage collection, I'm doubting a bit the numpy/core/src/multiarray/numpymemoryview.c implementation, since that's the only part in Numpy that supports that. Alternatively, just replace numpymemoryview.c with the attached one which has cyclic GC stripped, and see if you still get the crash. Cheers, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean.
Re: [Numpy-discussion] printing structured arrays
On Sat, Mar 6, 2010 at 8:35 AM, Gökhan Sever gokhanse...@gmail.com wrote: On Fri, Mar 5, 2010 at 8:00 AM, Bruce Schultz bruce.schu...@gmail.com wrote: Output is: ### ndarray [[ 1. 2. ] [ 3. 4.1]] ### structured array [(1.0, 2.0) (3.0, 4.0996)] Thanks Bruce I still couldn't figure out how floating point numbers look nicely on screen in cases like yours (i.e., trying numpy.array2string()) but you can make sure by using numpy.savetxt(file, array, fmt=%.1f) you will always have specified precision in the written file. Using numpy.array2string() gives the same format as the output above. Using numpy.savetxt() creates the same nicely formatted file containing the lines below for both structured and unstructured arrays. 1.0 2.0 3.0 4.1 But I was mainly curious about this because I just want to quickly dump data out to the console for debugging, and the unstructured format is obviously much easier to read. It seems like from other discussion in the thread that the quick solution is to convert back to a unstructured array with something like view((float, 2)), but that seems a bit clumsy. Cheers Bruce ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in signal.convolve2d? Alternative?
Hi, Just another update: signal.convolve and signal.fftconvolve indeed do not seem to have the problem, however, they are slower by at least a factor of 2 for my situation. Moreover, I also tried out the numpy 1.4.x branch and the latest scipy svn, and a short test seemed to indicate that the memory leak still was present (albeit, interestingly, memory usage grew much slower), but then something else stopped working in my main program, so I converted back to scipy 7.1 and numpy 1.3.0 for now. I might have confused things somewhere along the line, though, I'm not an expert with these things. Maybe other people can confirm the problem one way or another. Thanks, David On Tue, Mar 9, 2010 at 9:32 PM, josef.p...@gmail.com wrote: On Tue, Mar 9, 2010 at 4:24 PM, David Reichert d.p.reich...@sms.ed.ac.uk wrote: Hm, upgrading scipy from 0.7.0 to 0.7.1 didn't do the trick for me (still running numpy 1.3.0). I'm not sure if I feel confident enough to use developer versions, but I'll look into it. If you don't need the extra options, you could also use in the meantime the nd version signal.convolve or the fft version signal.fftconvolve Josef Cheers David On Tue, Mar 9, 2010 at 7:57 PM, Robert Kern robert.k...@gmail.com wrote: On Tue, Mar 9, 2010 at 13:49, David Reichert d.p.reich...@sms.ed.ac.uk wrote: Hi, I just reported a memory leak with matrices, and I might have found another (unrelated) one in the convolve2d function: import scipy.signal from numpy import ones while True: scipy.signal.convolve2d(ones((1,1)), ones((1,1))) This does not leak for me using SVN versions of numpy and scipy. I recommend upgrading. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PSF GSoC 2010 (Py3K focus)
josef.p...@gmail.com wrote: On Tue, Mar 9, 2010 at 9:02 AM, Pauli Virtanen pav...@iki.fi wrote: Mon, 08 Mar 2010 22:39:00 -0700, Charles R Harris wrote: On Mon, Mar 8, 2010 at 10:29 PM, Jarrod Millman mill...@berkeley.eduwrote: I added Titus' email regarding the PSF's focus on Py3K-related projects to our SoC ideas wiki page: http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas Given Titus' email, this is the most likely list of projects we will get accepted this year: - finish porting NumPy to Py3K I think Numpy is pretty much done. It needs use testing to wring out any small oddities, but it doesn't look to me like a GSOC project at the moment. Maybe Pauli can weigh in here. I think it's pretty much done. Even f2py should work. What's left is mostly testing it, and fixing any issues that crop up. Note that the SVN Numpy should preferably still see more testing on Python 2 against real-world packages that use it, before release. There's been a significant amount of code churn. - port SciPy to Py3K This project might be more appropriate, although I'm not clear on what needs to be done. I think porting Scipy proceeds in these steps: 1. Set up a similar 2to3-using build system for Python 3 as currently in Numpy. 2. Change the code so that it works both on Python 2 and Python 3. This can be done one submodule at a time. For C code, the changes required are mainly PyString usage. Some macros need to be defined to allow the same code to build on Py2 and Py3. Scipy is probably easier to port than Numpy here, since IIRC it doesn't mess much with strings, and its use of the Python C-API is much more limited. Also, parts written with Cython need essentially no porting. For Python code, the main issue is again bytes vs. unicode fixes, mainly inserting numpy.compat.asbytes() at correct locations to make e.g. bytes literals. All I/O code should be changed to work solely with Bytes. Since 2to3 takes care of most the other things, the remaining work is in fixing things it mishandles. I don't have a good estimate for how much work is in making Scipy work. I'd guess the work needed is probably slightly less than for Numpy. a few questions: Is scipy.special difficult or time consuming to port? I don't think it would be - most of it is fortran, so assuming f2py works correctly for py3k, there should not be big issues. In the build system, is it possible not to build some subpackages that might be slow in being ported, e.g. ndimage, weave? The way I used to do when porting to a new compiler/new platform is simply to comment everything but one package at a time in scipy/setup.py. linalg/lib/clusters are the first ones to port. I don't think special depends on much more than linalg/lib, but I could be wrong. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] crash at prompt exit after running test
more fun : [co...@jarrett tests]$ pwd /home/cohen/sources/python/numpy/numpy/core/tests [co...@jarrett tests]$ python -c 'import test_ufunc' python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Aborted (core dumped) What happens if you only import the umath_tests module (or something, it's a .so under numpy.core)? Just importing test_ufunc.py probably doesn't run a lot of extension code. Since you don't get crashes only by importing numpy, I really don't see many alternatives any more... Thanks, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PSF GSoC 2010 (Py3K focus)
On Tue, Mar 9, 2010 at 7:32 PM, David Cournapeau da...@silveregg.co.jp wrote: josef.p...@gmail.com wrote: On Tue, Mar 9, 2010 at 9:02 AM, Pauli Virtanen pav...@iki.fi wrote: Mon, 08 Mar 2010 22:39:00 -0700, Charles R Harris wrote: On Mon, Mar 8, 2010 at 10:29 PM, Jarrod Millman mill...@berkeley.eduwrote: I added Titus' email regarding the PSF's focus on Py3K-related projects to our SoC ideas wiki page: http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas Given Titus' email, this is the most likely list of projects we will get accepted this year: - finish porting NumPy to Py3K I think Numpy is pretty much done. It needs use testing to wring out any small oddities, but it doesn't look to me like a GSOC project at the moment. Maybe Pauli can weigh in here. I think it's pretty much done. Even f2py should work. What's left is mostly testing it, and fixing any issues that crop up. Note that the SVN Numpy should preferably still see more testing on Python 2 against real-world packages that use it, before release. There's been a significant amount of code churn. - port SciPy to Py3K This project might be more appropriate, although I'm not clear on what needs to be done. I think porting Scipy proceeds in these steps: 1. Set up a similar 2to3-using build system for Python 3 as currently in Numpy. 2. Change the code so that it works both on Python 2 and Python 3. This can be done one submodule at a time. For C code, the changes required are mainly PyString usage. Some macros need to be defined to allow the same code to build on Py2 and Py3. Scipy is probably easier to port than Numpy here, since IIRC it doesn't mess much with strings, and its use of the Python C-API is much more limited. Also, parts written with Cython need essentially no porting. For Python code, the main issue is again bytes vs. unicode fixes, mainly inserting numpy.compat.asbytes() at correct locations to make e.g. bytes literals. All I/O code should be changed to work solely with Bytes. Since 2to3 takes care of most the other things, the remaining work is in fixing things it mishandles. I don't have a good estimate for how much work is in making Scipy work. I'd guess the work needed is probably slightly less than for Numpy. a few questions: Is scipy.special difficult or time consuming to port? I don't think it would be - most of it is fortran, so assuming f2py works correctly for py3k, there should not be big issues. In the build system, is it possible not to build some subpackages that might be slow in being ported, e.g. ndimage, weave? The way I used to do when porting to a new compiler/new platform is simply to comment everything but one package at a time in scipy/setup.py. linalg/lib/clusters are the first ones to port. I don't think special depends on much more than linalg/lib, but I could be wrong. Thanks, Josef David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in signal.convolve2d? Alternative?
On Tue, Mar 9, 2010 at 7:13 PM, David Reichert d.p.reich...@sms.ed.ac.uk wrote: Hi, Just another update: signal.convolve and signal.fftconvolve indeed do not seem to have the problem, however, they are slower by at least a factor of 2 for my situation. Moreover, I also tried out the numpy 1.4.x branch and the latest scipy svn, and a short test seemed to indicate that the memory leak still was present (albeit, interestingly, memory usage grew much slower), but then something else stopped working in my main program, so I converted back to scipy 7.1 and numpy 1.3.0 for now. I might have confused things somewhere along the line, though, I'm not an expert with these things. Maybe other people can confirm the problem one way or another. I ran your convolve2d example for a few minutes without any increase in memory with numpy 1.4.0 and scipy svn that is about 2 months old Josef Thanks, David On Tue, Mar 9, 2010 at 9:32 PM, josef.p...@gmail.com wrote: On Tue, Mar 9, 2010 at 4:24 PM, David Reichert d.p.reich...@sms.ed.ac.uk wrote: Hm, upgrading scipy from 0.7.0 to 0.7.1 didn't do the trick for me (still running numpy 1.3.0). I'm not sure if I feel confident enough to use developer versions, but I'll look into it. If you don't need the extra options, you could also use in the meantime the nd version signal.convolve or the fft version signal.fftconvolve Josef Cheers David On Tue, Mar 9, 2010 at 7:57 PM, Robert Kern robert.k...@gmail.com wrote: On Tue, Mar 9, 2010 at 13:49, David Reichert d.p.reich...@sms.ed.ac.uk wrote: Hi, I just reported a memory leak with matrices, and I might have found another (unrelated) one in the convolve2d function: import scipy.signal from numpy import ones while True: scipy.signal.convolve2d(ones((1,1)), ones((1,1))) This does not leak for me using SVN versions of numpy and scipy. I recommend upgrading. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] crash at prompt exit after running test
On Tue, Mar 9, 2010 at 5:52 PM, Johann Cohen-Tanugi co...@lpta.in2p3.frwrote: more fun : [co...@jarrett tests]$ pwd /home/cohen/sources/python/numpy/numpy/core/tests [co...@jarrett tests]$ python -c 'import test_ufunc' python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Aborted (core dumped) and in the debugger: (gdb) run Starting program: /usr/bin/python warning: .dynamic section for /lib/libpthread.so.0 is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for /lib/libdl.so.2 is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for /lib/libc.so.6 is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations [Thread debugging using libthread_db enabled] Python 2.6.2 (r262:71600, Jan 25 2010, 18:46:45) [GCC 4.4.2 20091222 (Red Hat 4.4.2-20)] on linux2 Type help, copyright, credits or license for more information. import test_ufunc python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Program received signal SIGABRT, Aborted. 0x00aab416 in __kernel_vsyscall () Missing separate debuginfos, use: debuginfo-install atlas-3.8.3-12.fc12.i686 libgcc-4.4.3-4.fc12.i686 libgfortran-4.4.3-4.fc12.i686 (gdb) bt #0 0x00aab416 in __kernel_vsyscall () #1 0x00159a91 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #2 0x0015b35a in abort () at abort.c:92 #3 0x00152be8 in __assert_fail (assertion=value optimized out, file=value optimized out, line=value optimized out, function=value optimized out) at assert.c:81 #4 0x0050931e in visit_decref (op=value optimized out, data=value optimized out) at Modules/gcmodule.c:277 #5 0x0047c8c2 in dict_traverse (op=value optimized out, visit=value optimized out, arg=value optimized out) at Objects/dictobject.c:2003 #6 0x00509af3 in subtract_refs (generation=value optimized out) at Modules/gcmodule.c:296 #7 collect (generation=value optimized out) at Modules/gcmodule.c:817 #8 0x0050a640 in PyGC_Collect () at Modules/gcmodule.c:1292 #9 0x004fb0f0 in Py_Finalize () at Python/pythonrun.c:424 #10 0x0050868f in Py_Main (argc=value optimized out, argv=value optimized out) at Modules/main.c:625 #11 0x080485c8 in main (argc=value optimized out, argv=value optimized out) at Modules/python.c:23 which looks identical to the bt I sent to Robert earlier on. HTH, Johann On 03/10/2010 12:43 AM, Johann Cohen-Tanugi wrote: On 03/10/2010 12:33 AM, Johann Cohen-Tanugi wrote: On 03/10/2010 12:07 AM, Pauli Virtanen wrote: ti, 2010-03-09 kello 21:14 +0100, Johann Cohen-Tanugi kirjoitti: thinking about it, this morning there was a fedora update to python, so I am using 2.6.2-4.fc12. Looks like the problem is in python itself, hence this piece of info. That the problem would be in Python is not so clear to me. Can you try running it with the previous Python shipped by Fedora? Do you see the problem then? What's the previous version, btw? 2.6.2-1 IIRC. I would have to check, and I am not sure how to either query this information or step back one update up with yum :( Memory errors are somewhat difficult to debug. Can you try running only a certain subset of the tests, first nosetests numpy.core crash Be sure to set Pythonpath so that you get the correct Numpy version. If it segfaults, proceed to (under numpy/core/tests) nosetests test_multiarray.py nosetests test_multiarray.py:TestNewBufferProtocol neither crash, so the problem is not there I followed your lead and tried each script and ended up with : [co...@jarrett tests]$ nosetests test_ufunc.py . -- Ran 13 tests in 1.146s OK python: Modules/gcmodule.c:277: visit_decref: Assertion `gc-gc.gc_refs != 0' failed. Aborted (core dumped) so test_ufunc.py seems to be at stake Since the crash occurs in cyclic garbage collection, I'm doubting a bit the numpy/core/src/multiarray/numpymemoryview.c implementation, since that's the only part in Numpy that supports that. Alternatively, just replace numpymemoryview.c with the attached one which has cyclic GC stripped, and see if you still get the crash. Cheers, Pauli ___ NumPy-Discussion mailing listnumpy-discuss...@scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion -- This message has been scanned for viruses and dangerous content by *MailScanner* http://www.mailscanner.info/, and is believed to be clean. ___ NumPy-Discussion mailing listnumpy-discuss...@scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion Python 2.6.2 is rather old by now, the bugfix releases
