Re: [Numpy-discussion] [Swig-user] swig numpy2carray converters
These corrections have been committed. Thanks. On Dec 1, 2007, at 9:21 AM, Georg Holzmann wrote: >> * A new ARGOUTVIEW suite of typemaps is provided that allows your >> wrapped function >>to provide a pointer to internal data and that returns a numpy >> array encapsulating >>it. > > Thanks for integrating it ! > >> * New typemaps are provided that correctly handle FORTRAN ordered 2D >> and 3D arrays. > > I have some problem with your FORTRAN implementation. > The problem is how you set the flags in numpy.i "int > require_fortran(PyArrayObject* ary)" (~ line 402). > > You do it like this: > ary->flags = ary->flags | NPY_F_CONTIGUOUS; > which does not work (at least on my computer) - I still get usual > C-ordered arrays returned. > > However, it does work if you set the flags like this: > ary->flags = NPY_FARRAY; > >> Tests for the ARGOUTVIEW and FORTRAN ordered arrays have also been >> added, and the documentation (doc/numpy_swig.*) has been updated to >> reflect all of these changes. > > A small typo: in the docs you also write about 1D FORTRAN ARGOUTVIEW > wrappers: > > ( DATA_TYPE** ARGOUTVIEW_FARRAY1, DIM_TYPE* DIM1 ) > ( DIM_TYPE* DIM1, DATA_TYPE** ARGOUTVIEW_FARRAY1 ) > > which of course do not exist. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370Email: [EMAIL PROTECTED] ** ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] display numpy array as image (Giorgio F. Gilestro)
On Freitag 30 November 2007, Joe Harrington wrote: > I was misinformed about the status of numdisplay's pages. The package > is available as both part of stsci_python and independently, and its > (up-to-date) home page is here: > > http://stsdas.stsci.edu/numdisplay/ I had a look at ds9/numdisplay, and as a summary, I found a nice viewer for scalar images with colormap support, and x/y projections. What I missed though is the display of RGB images. It looks as if ds9 was capable of doing so (I could add "frames" of RGB type), but I did not find a way to feed them with data. numdisplay.display(myarray) seems to only support 2D-arrays. Ciao, / /.o. /--/ ..o / / ANS ooo signature.asc Description: This is a digitally signed message part. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Loading a > GB file into array
On Samstag 01 Dezember 2007, Martin Spacek wrote: > Kurt Smith wrote: > > You might try numpy.memmap -- others have had success with it for > > large files (32 bit should be able to handle a 1.3 GB file, AFAIK). > > Yeah, I looked into numpy.memmap. Two issues with that. I need to > eliminate as much disk access as possible while my app is running. I'm > displaying stimuli on a screen at 200Hz, so I have up to 5ms for each > movie frame to load before it's too late and it drops a frame. I'm sort > of faking a realtime OS on windows by setting the process priority > really high. Disk access in the middle of that causes frames to drop. So > I need to load the whole file into physical RAM, although it need not be > contiguous. memmap doesn't do that, it loads on the fly as you index > into the array, which drops frames, so that doesn't work for me. Sounds as if using memmap and then copying each frame into a separate in-memory ndarray could help? Ciao, / /.o. /--/ ..o / / ANS ooo signature.asc Description: This is a digitally signed message part. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] swig numpy2carray converters
Hallo! > * A new ARGOUTVIEW suite of typemaps is provided that allows your > wrapped function >to provide a pointer to internal data and that returns a numpy > array encapsulating >it. Thanks for integrating it ! > * New typemaps are provided that correctly handle FORTRAN ordered 2D > and 3D arrays. I have some problem with your FORTRAN implementation. The problem is how you set the flags in numpy.i "int require_fortran(PyArrayObject* ary)" (~ line 402). You do it like this: ary->flags = ary->flags | NPY_F_CONTIGUOUS; which does not work (at least on my computer) - I still get usual C-ordered arrays returned. However, it does work if you set the flags like this: ary->flags = NPY_FARRAY; > Tests for the ARGOUTVIEW and FORTRAN ordered arrays have also been > added, and the documentation (doc/numpy_swig.*) has been updated to > reflect all of these changes. A small typo: in the docs you also write about 1D FORTRAN ARGOUTVIEW wrappers: ( DATA_TYPE** ARGOUTVIEW_FARRAY1, DIM_TYPE* DIM1 ) ( DIM_TYPE* DIM1, DATA_TYPE** ARGOUTVIEW_FARRAY1 ) which of course do not exist. LG Georg ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Loading a > GB file into array
Ivan Vilata i Balaguer (el 2007-11-30 a les 19:19:38 +0100) va dir:: > Well, one thing you could do is dump your data into a PyTables_ > ``CArray`` dataset, which you may afterwards access as if its was a > NumPy array to get slices which are actually NumPy arrays. PyTables > datasets have no problem in working with datasets exceeding memory size. >[...] I've put together the simple script I've attached which dumps a binary file into a PyTables' ``CArray`` or loads it to measure the time taken to load each frame. I've run it on my laptop, which has a not very fast 4200 RPM laptop hard disk, and I've reached average times of 16 ms per frame, after dropping caches with:: # sync && echo 1 > /proc/sys/vm/drop_caches This I've done with the standard chunkshape and no compression. Your data may lean itself very well to bigger chunkshapes and compression, which should lower access times even further. Since (as David pointed out) 200 Hz may be a little exaggerated for human eye, loading individual frames from disk may prove more than enough for your problem. HTH, :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ Cárabos Coop. V. V V Enjoy Data "" from __future__ import with_statement from time import time from contextlib import nested import numpy as np from tables import openFile, UInt8Atom, Filters width, height = 640, 480 # 300 KiB per (greyscale) frame def dump_frames_1(npfname, h5fname, nframes): """Dump `nframes` frames to a ``CArray`` dataset.""" with nested(file(npfname, 'rb'), openFile(h5fname, 'w')) as (npf, h5f): frames = h5f.createCArray( '/', 'frames', atom=UInt8Atom(), shape=(nframes, height, width), chunkshape=(1, height/2, width), # filters=Filters(complib='lzo'), ) framesize = width * height * 1 for framei in xrange(nframes): frame = np.fromfile(npf, np.uint8, count=framesize) frame.shape = (height, width) frames[framei] = frame def dump_frames_2(npfname, h5fname, nframes): """Dump `nframes` frames to an ``EArray`` dataset.""" with nested(file(npfname, 'rb'), openFile(h5fname, 'w')) as (npf, h5f): frames = h5f.createEArray( '/', 'frames', atom=UInt8Atom(), shape=(0, height, width), expectedrows=nframes, # chunkshape=(1, height/2, width), # filters=Filters(complib='lzo'), ) framesize = width * height * 1 for framei in xrange(nframes): frame = np.fromfile(npf, np.uint8, count=framesize) frame.shape = (1, height, width) frames.append(frame) def load_frames(h5fname): with openFile(h5fname, 'r') as h5f: frames = h5f.root.frames nframes = len(frames) times = np.zeros(nframes, float) for framei in xrange(nframes): t0 = time() frame = frames[framei] t1 = time() times[framei] = t1 - t0 print ( "Load times for %d frames: min=%.4f avg=%.4f max=%.4f" % (nframes, np.min(times), np.average(times), np.max(times)) ) if __name__ == '__main__': import sys if sys.argv[1] == 'dump': npfname, h5fname, nframes = sys.argv[2:] nframes = int(nframes, 10) dump_frames_1(npfname, h5fname, nframes) elif sys.argv[1] == 'load': load_frames(sys.argv[2]) else: print >> sys.stderr, """\ Usage:
Re: [Numpy-discussion] Loading a > GB file into array
On Dec 1, 2007 12:09 AM, Martin Spacek <[EMAIL PROTECTED]> wrote: > Kurt Smith wrote: > > You might try numpy.memmap -- others have had success with it for > > large files (32 bit should be able to handle a 1.3 GB file, AFAIK). > > Yeah, I looked into numpy.memmap. Two issues with that. I need to > eliminate as much disk access as possible while my app is running. I'm > displaying stimuli on a screen at 200Hz, so I have up to 5ms for each > movie frame to load before it's too late and it drops a frame. I'm sort > of faking a realtime OS on windows by setting the process priority > really high. Disk access in the middle of that causes frames to drop. So > I need to load the whole file into physical RAM, although it need not be > contiguous. memmap doesn't do that, it loads on the fly as you index > into the array, which drops frames, so that doesn't work for me. > > The 2nd problem I had with memmap was that I was getting a WindowsError > related to memory: > > >>> data = np.memmap(1.3GBfname, dtype=np.uint8, mode='r') > > Traceback (most recent call last): > File "", line 1, in > File "C:\bin\Python25\Lib\site-packages\numpy\core\memmap.py", line > 67, in __new__ > mm = mmap.mmap(fid.fileno(), bytes, access=acc) > WindowsError: [Error 8] Not enough storage is available to process this > command > > > This was for the same 1.3GB file. This is different from previous memory > errors I mentioned. I don't get this on ubuntu. I can memmap a file up > to 2GB on ubuntu no problem, but any larger than that and I get this: > > >>> data = np.memmap(2.1GBfname, dtype=np.uint8, mode='r') > > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python2.5/site-packages/numpy/core/memmap.py", line > 67, in __new__ > mm = mmap.mmap(fid.fileno(), bytes, access=acc) > OverflowError: cannot fit 'long' into an index-sized integer > > The OverflowError is on the bytes argument. If I try doing the mmap.mmap > directly in Python, I get the same error. So I guess it's due to me > running 32bit ubuntu. > Hi, reading this thread I have two comments. a) *Displaying* at 200Hz probably makes little sense, since humans would only see about max. of 30Hz (aka video frame rate). Consequently you would want to separate your data frame rate, that (as I understand) you want to save data to disk and - asynchrounously - "display as many frames as you can" (I have used pyOpenGL for this with great satisfaction) b) To my knowledge, any OS Linux, Windows an OSX can max. allocate about 1GB of data - assuming you have a 32 bit machine. The actual numbers I measured varied from about 700MB to maybe 1.3GB. In other words, you would be right at the limit. (For 64bit, you would have to make sure ALL parts are 64bit, e.g. The python version must be >=2.5, python must have been compiled using a 64-bit compiler *and* the windows version (XP-64)) This holds true the same for physical ram allocation and for memmap allocation. My solution to this was to "wait" for the 64bit not tested yet ;-) Cheers, Sebastian Haase ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Loading a > GB file into array
Martin Spacek wrote: > Kurt Smith wrote: > > You might try numpy.memmap -- others have had success with it for > > large files (32 bit should be able to handle a 1.3 GB file, AFAIK). > > Yeah, I looked into numpy.memmap. Two issues with that. I need to > eliminate as much disk access as possible while my app is running. I'm > displaying stimuli on a screen at 200Hz, so I have up to 5ms for each > movie frame to load before it's too late and it drops a frame. I'm sort > of faking a realtime OS on windows by setting the process priority > really high. Disk access in the middle of that causes frames to drop. So > I need to load the whole file into physical RAM, although it need not be > contiguous. memmap doesn't do that, it loads on the fly as you index > into the array, which drops frames, so that doesn't work for me. If you want to do it 'properly', it will be difficult, specially in python, specially on windows. This looks really similar to the problem of direct to disk recording, that is you record audio signals from the soundcard into the hard-drive (think recording a concert), and the proper design, at least on linux and mac os X, is to have several threads, one for the IO, one for any computation you may want to do which do not block on any condition, etc... and use special OS facilities (FIFO scheduling, lock pages into physical ram, etc...) as well as some special construct (lock-free ring buffers). This design works relatively well for musical applications, where the data has the same order of magnitude than what you are talking about, and the same kind of latency order (a few ms). This may be overkill for your application, though. > > The 2nd problem I had with memmap was that I was getting a WindowsError > related to memory: > > >>> data = np.memmap(1.3GBfname, dtype=np.uint8, mode='r') > > Traceback (most recent call last): >File "", line 1, in >File "C:\bin\Python25\Lib\site-packages\numpy\core\memmap.py", line > 67, in __new__ > mm = mmap.mmap(fid.fileno(), bytes, access=acc) > WindowsError: [Error 8] Not enough storage is available to process this > command > > > This was for the same 1.3GB file. This is different from previous memory > errors I mentioned. I don't get this on ubuntu. I can memmap a file up > to 2GB on ubuntu no problem, but any larger than that and I get this: > > >>> data = np.memmap(2.1GBfname, dtype=np.uint8, mode='r') > > Traceback (most recent call last): >File "", line 1, in >File "/usr/lib/python2.5/site-packages/numpy/core/memmap.py", line > 67, in __new__ > mm = mmap.mmap(fid.fileno(), bytes, access=acc) > OverflowError: cannot fit 'long' into an index-sized integer > > The OverflowError is on the bytes argument. If I try doing the mmap.mmap > directly in Python, I get the same error. So I guess it's due to me > running 32bit ubuntu. Yes. 32 bits means several things in this context: you have 32 bits for the virtual address space, but part of it is reserved for the kernel: this is configurable on linux, and there is a switch for this on windows. By default, on windows, it is split into half: 2 Gb for the kernel, 2 Gb for userspace. On linux, it depends on the distribution(on ubuntu, it seems that the default is 3G for user space, 1 G for kernel space, by reading the build config). I think part of the problem (memory error) is related to this, at least on windows. But the error you get above is easier to understand: an integer is 32 bits, but since it is signed, you cannot address more than 2^31 different locations with an integer. That's why with standard (ansi C stlib) functions related to files, you cannot access more than 2Gb; you need special api for that. In your case, because you cannot code your value with a signed 32 bits integer, you get this error, I guess (index-sized integer means signed integer, I guess). But even if it succeeded, you would be caught by the above problem (if you only have 2 Gb user space for virtual adressing, I don't think you can do a mmap with a size which is more than that, since the whole mapping is done at once; I am not so knowledgeable about OS, though, so I may be totally wrong on this). cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
