[Numpy-discussion] Call to mrecords users
All, I just commited some updates on mrecords (numpy.maskedarray branch). Overall, it's a reorganization/simplification of the code. Regular masked arrays can already recognize named fields, but the mask works only at the record level (ie, all the fields of one record are masked). In comparison, masked record arrays (mrecarrays) permit the masking of individual fields. The conversion between a masked array of records and a mrecarray is as easy as for the classical ndarray: just use a .view(mrecarray). I'd be grateful if the current users of mrecords could give the new version a try, and let me know whether everything works seamlessly or if new modifications need to be implemented. Thanks a lot in advance. PS: that goes as well for users of trecords (in scikits.timeseries): the package has been updated to match the recent modifications on mrecords ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [F2PY]: Allocatable Arrays
Pearu Peterson wrote: > On Fri, February 1, 2008 8:39 pm, Robert Kern wrote: >> Pearu Peterson wrote: >>> On Fri, February 1, 2008 1:28 pm, Andrea Gavana wrote: Hi All, I sent a couple of messages to f2py mailing list, but it seems like my problem has no simple solution so I thought to ask for some suggestions here. >>> Sorry, I haven't been around there long time. >> Are you going to continue not reading the f2py list? If so, you should >> point everyone there to this list and close the list. > > It is a bit embarrassing, I haven't read the list because I lost > the link to f2py list archives that I used to use in my mailing box > in the server.. but I have been also busy with non-python stuff > last years (I moved and got married..:). > Anyway, I have subscribed to the f2py list again I'll try to respond > to any messages that have unresolved issues, also in the arhives. Great. Thank you. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [F2PY]: Allocatable Arrays
On Fri, February 1, 2008 8:39 pm, Robert Kern wrote: > Pearu Peterson wrote: >> On Fri, February 1, 2008 1:28 pm, Andrea Gavana wrote: >>> Hi All, >>> >>> I sent a couple of messages to f2py mailing list, but it seems >>> like my problem has no simple solution so I thought to ask for some >>> suggestions here. >> >> Sorry, I haven't been around there long time. > > Are you going to continue not reading the f2py list? If so, you should > point everyone there to this list and close the list. It is a bit embarrassing, I haven't read the list because I lost the link to f2py list archives that I used to use in my mailing box in the server.. but I have been also busy with non-python stuff last years (I moved and got married..:). Anyway, I have subscribed to the f2py list again I'll try to respond to any messages that have unresolved issues, also in the arhives. Thanks, Pearu ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [F2PY]: Allocatable Arrays
Pearu Peterson wrote: > On Fri, February 1, 2008 1:28 pm, Andrea Gavana wrote: >> Hi All, >> >> I sent a couple of messages to f2py mailing list, but it seems >> like my problem has no simple solution so I thought to ask for some >> suggestions here. > > Sorry, I haven't been around there long time. Are you going to continue not reading the f2py list? If so, you should point everyone there to this list and close the list. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Can not update a submatrix
A Thursday 31 January 2008, Francesc Altet escrigué: > A Wednesday 30 January 2008, Timothy Hochberg escrigué: > > [...a fine explanation by Anne and Timothy...] > > Ok. As it seems that this subject has interest enough, I went ahead > and created a small document about views vs copies at: > > http://www.scipy.org/Cookbook/ViewsVsCopies Ooops, I think I've missed the NumPy tutorial: http://www.scipy.org/Tentative_NumPy_Tutorial which already talked about copies vs views :-/. Well, I think my small document can complement some parts of the tutorial. I'll do that as soon as I can and remove the recipe from the cookbook. Sorry for the noise. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-" ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] histogramdd memory needs
Hi Lars, [...] 2008/2/1, Lars Friedrich <[EMAIL PROTECTED]>: > > > 1) How can I tell histogramdd to use another dtype than float64? My bins > will be very little populated so an int16 should be sufficient. Without > normalization, a Integer dtype makes more sense to me. There is no way you'll be able to ask that without tweaking the histogramdd function yourself. The relevant bit of code is the instantiation of hist : hist = zeros(nbin.prod(), float) 2) Is there a way to use another algorithm (at the cost of performance) > that uses less memory during calculation so that I can generate bigger > histograms? You could work through your array block by block. Simply fix the range and generate an histogram for each slice of 100k data and sum them up at the end. The current histogram and histogramdd implementation has the advantage of being general, that is you can work with uniform or non-uniform bins, but it is not particularly efficient, at least for large number of bins (>30). Cheers, David My numpy version is '1.0.4.dev3937' > > Thanks, > Lars > > > -- > Dipl.-Ing. Lars Friedrich > > Photonic Measurement Technology > Department of Microsystems Engineering -- IMTEK > University of Freiburg > Georges-Köhler-Allee 102 > D-79110 Freiburg > Germany > > phone: +49-761-203-7531 > fax: +49-761-203-7537 > room: 01 088 > email: [EMAIL PROTECTED] > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [F2PY]: Allocatable Arrays
On Fri, February 1, 2008 4:18 pm, Andrea Gavana wrote: > Hi Lisandro, > > On Feb 1, 2008 1:59 PM, Lisandro Dalcin wrote: >> Sorry if I'm making noise, my knowledge of fortran is really little, >> but in your routine AllocateDummy your are fist allocating and next >> deallocating the arrays. Are you sure you can then access the contents >> of your arrays after deallocating them? > > Thank you for your answer. > > Unfortunately it seems that it doesn't matter whether I deallocate > them or not, I still get the compilation warning and I can't access > those variable in any case. You cannot access those becase they are deallocated. Try to disable deallocate statements in your fortran code. > It seems like f2py (or python or whatever) > does not like having more than 1 allocatable array inside a MODULE > declaration. This is not true. >> How much complicated is your binary format? > > *Very* complex. The fact is, I already know how to read those files in > Fortran, is the linking with Python via f2py that is driving me mad. I > can't believe no one has used before allocatable arrays as outputs > (whether from a subroutine or from a module). You can use allocatable arrays from module in Python as described in f2py users guide. It could be that the problem is related to deallocating the arrays in the fortran code. Regards, Pearu ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [F2PY]: Allocatable Arrays
On Fri, February 1, 2008 1:28 pm, Andrea Gavana wrote:
> Hi All,
>
> I sent a couple of messages to f2py mailing list, but it seems
> like my problem has no simple solution so I thought to ask for some
> suggestions here.
Sorry, I haven't been around there long time.
> Basically, I read some huge unformatted binary files which contain
> time-step data from a reservoir simulation. I don't know the
> dimensions (i.e., lengths) of the vectors I am going to read, and I
> find out this information only when I start reading the file. So, I
> thought it would be nice to do something like:
>
> 1) Declare outputVector as allocatable;
> 2) Start reading the file;
> 3) Find the outputVector dimension and allocate it;
> 4) Read the data in the outputVector;
looks ok.
> 5) Return this outputVector.
What do you mean by "return"? You cannot return allocatable arrays
as far as comes to using f2py for generating wrappers. However,
you can access allocatable array outputVector if it is module data,
as you do below.
> It works when I compile it and build it in Fortran as an executable
> (defining a "main" program in my f90 module), but it bombs when I try
> to use it from Python with the error:
>
> C:\Documents and Settings\gavana\Desktop\ECLIPSEReader>prova.py
> Traceback (most recent call last):
> File "C:\Documents and
> Settings\gavana\Desktop\ECLIPSEReader\prova.py", line 3, in
>inteHead, propertyNames, propertyTypes, propertyNumbers =
> ECLIPSEReader.init.readinspec("OPT_INJ.INSPEC")
> ValueError: failed to create intent(cache|hide)|optional array-- must
> have defined dimensions but got (-1,)
This exception is not directly related to what follows below.
> So, I have tried with a suggestion given in the f2py mailing list, and
> I found out that this routine works:
>
>
> MODULE DUMMY
> IMPLICIT NONE
>
> ! Ok, so I want an allocatable array as output
>
> real(8), allocatable :: realOutput(:)
...
> END MODULE DUMMY
>
>
> But this one doesn't work:
>
>
> MODULE DUMMY
> IMPLICIT NONE
>
> ! Ok, so I want an allocatable array as output
>
> real(8), allocatable :: realOutput(:)
> integer, allocatable :: inteOutput(:)
>
> CONTAINS
...
> END MODULE DUMMY
This one works fine here:
$ f2py -c -m m2 m2.f90 --fcompiler=gnu95
>>> import m2
>>> print m2.dummy.__doc__
realoutput - 'd'-array(-1), not allocated
inteoutput - 'i'-array(-1), not allocated
allocatedummy - Function signature:
allocatedummy(dummyinput)
Required arguments:
dummyinput : input int
> The difference between the 2 scripts, is just that in the second one I
> want 2 allocatable arrays instead of 1. When I compile it with f2py, I
> get this warning from getarrdims:
>
> Building modules...
>Building module "dummy"...
>Constructing F90 module support for "dummy"...
> Variables: realoutput inteoutput
^
Looks like both allocatable arrays should be present also in your
case.
> getarrdims:warning: assumed shape array, using 0 instead of ':'
> getarrdims:warning: assumed shape array, using 0 instead of ':'
These warnings can be ignored.
> Which is not present if I compile script number 1. Actually, if I run
> script 2, I can't access anymore the 2 variables realoutput and
> inteoutput (they are not there), while with script 1 I can easily
> access realoutput by writing dummy.dummy.realoutput.
What do you mean by accessing? What happens when you type:
dummy.dummy.inteoutput
?
Note that when you call AllocateDummy function, then you allocate
and then deallocate the arrays. So, in Python dummy.dummy.realoutput
and dummy.dummy.inteoutput should always return None.
See
http://cens.ioc.ee/projects/f2py2e/usersguide/index.html#allocatable-arrays
for how to use allocatable module data from Python.
> I can't actually see any big difference between the 2 scripts... am I
> missing something?
>
> This is Windows XP, Python 2.5, numpy 1.0.3.1, Compaq Visual Fortran
> 6.6, MS Visual Studio .NET 2003.
I am using numpy from svn.
Regards,
Pearu
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [F2PY]: Allocatable Arrays
Hi Lisandro,
On Feb 1, 2008 1:59 PM, Lisandro Dalcin wrote:
> Sorry if I'm making noise, my knowledge of fortran is really little,
> but in your routine AllocateDummy your are fist allocating and next
> deallocating the arrays. Are you sure you can then access the contents
> of your arrays after deallocating them?
Thank you for your answer.
Unfortunately it seems that it doesn't matter whether I deallocate
them or not, I still get the compilation warning and I can't access
those variable in any case. It seems like f2py (or python or whatever)
does not like having more than 1 allocatable array inside a MODULE
declaration.
> How much complicated is your binary format?
*Very* complex. The fact is, I already know how to read those files in
Fortran, is the linking with Python via f2py that is driving me mad. I
can't believe no one has used before allocatable arrays as outputs
(whether from a subroutine or from a module).
> On 2/1/08, Andrea Gavana <[EMAIL PROTECTED]> wrote:
> > Hi All,
> >
> > I sent a couple of messages to f2py mailing list, but it seems
> > like my problem has no simple solution so I thought to ask for some
> > suggestions here.
> >
> > Basically, I read some huge unformatted binary files which contain
> > time-step data from a reservoir simulation. I don't know the
> > dimensions (i.e., lengths) of the vectors I am going to read, and I
> > find out this information only when I start reading the file. So, I
> > thought it would be nice to do something like:
> >
> > 1) Declare outputVector as allocatable;
> > 2) Start reading the file;
> > 3) Find the outputVector dimension and allocate it;
> > 4) Read the data in the outputVector;
> > 5) Return this outputVector.
> >
> > It works when I compile it and build it in Fortran as an executable
> > (defining a "main" program in my f90 module), but it bombs when I try
> > to use it from Python with the error:
> >
> > C:\Documents and Settings\gavana\Desktop\ECLIPSEReader>prova.py
> > Traceback (most recent call last):
> > File "C:\Documents and
> > Settings\gavana\Desktop\ECLIPSEReader\prova.py", line 3, in
> >inteHead, propertyNames, propertyTypes, propertyNumbers =
> > ECLIPSEReader.init.readinspec("OPT_INJ.INSPEC")
> > ValueError: failed to create intent(cache|hide)|optional array-- must
> > have defined dimensions but got (-1,)
> >
> >
> > So, I have tried with a suggestion given in the f2py mailing list, and
> > I found out that this routine works:
> >
> >
> > MODULE DUMMY
> > IMPLICIT NONE
> >
> > ! Ok, so I want an allocatable array as output
> >
> > real(8), allocatable :: realOutput(:)
> >
> > CONTAINS
> >
> >subroutine AllocateDummy(dummyInput)
> >
> >implicit none
> >save
> >
> >! dummyInput is *not* used, it's here just as
> >! an example
> >integer, intent(in) :: dummyInput
> >
> >! Allocate and build the output array
> >allocate(realOutput(10))
> >
> >realOutput(1:10) = 0.0
> >realOutput(3) = 3.0
> >realOutput(7) = 7.0
> >
> >deallocate(realOutput)
> >
> >return
> >
> >end subroutine AllocateDummy
> >
> >
> > END MODULE DUMMY
> >
> >
> >
> > But this one doesn't work:
> >
> >
> > MODULE DUMMY
> > IMPLICIT NONE
> >
> > ! Ok, so I want an allocatable array as output
> >
> > real(8), allocatable :: realOutput(:)
> > integer, allocatable :: inteOutput(:)
> >
> > CONTAINS
> >
> >subroutine AllocateDummy(dummyInput)
> >
> >implicit none
> >save
> >
> >! dummyInput is *not* used, it's here just as
> >! an example
> >integer, intent(in) :: dummyInput
> >
> >
> >! Allocate and build the output array
> >allocate(realOutput(10))
> >allocate(inteOutput(20))
> >
> >realOutput(1:10) = 0.0
> >realOutput(3) = 3.0
> >realOutput(7) = 7.0
> >
> >inteOutput(10) = 2
> >
> >deallocate(realOutput)
> >deallocate(inteOutput)
> >
> >return
> >
> >end subroutine AllocateDummy
> >
> >
> > END MODULE DUMMY
> >
> >
> >
> > The difference between the 2 scripts, is just that in the second one I
> > want 2 allocatable arrays instead of 1. When I compile it with f2py, I
> > get this warning from getarrdims:
> >
> > Building modules...
> >Building module "dummy"...
> >Constructing F90 module support for "dummy"...
> > Variables: realoutput inteoutput
> > getarrdims:warning: assumed shape array, using 0 instead of ':'
> > getarrdims:warning: assumed shape array, using 0 instead of ':'
> >Constructing wrapper function
> > "dummy.allocatedummy"...
> > allocatedummy(dummyinput)
> >
> >
> > Which is not present if
Re: [Numpy-discussion] [F2PY]: Allocatable Arrays
Sorry if I'm making noise, my knowledge of fortran is really little,
but in your routine AllocateDummy your are fist allocating and next
deallocating the arrays. Are you sure you can then access the contents
of your arrays after deallocating them?
How much complicated is your binary format? For simple formats, you
can just use numpy to read binary data, I use this sometimes, but
again, for simple formats.
On 2/1/08, Andrea Gavana <[EMAIL PROTECTED]> wrote:
> Hi All,
>
> I sent a couple of messages to f2py mailing list, but it seems
> like my problem has no simple solution so I thought to ask for some
> suggestions here.
>
> Basically, I read some huge unformatted binary files which contain
> time-step data from a reservoir simulation. I don't know the
> dimensions (i.e., lengths) of the vectors I am going to read, and I
> find out this information only when I start reading the file. So, I
> thought it would be nice to do something like:
>
> 1) Declare outputVector as allocatable;
> 2) Start reading the file;
> 3) Find the outputVector dimension and allocate it;
> 4) Read the data in the outputVector;
> 5) Return this outputVector.
>
> It works when I compile it and build it in Fortran as an executable
> (defining a "main" program in my f90 module), but it bombs when I try
> to use it from Python with the error:
>
> C:\Documents and Settings\gavana\Desktop\ECLIPSEReader>prova.py
> Traceback (most recent call last):
> File "C:\Documents and
> Settings\gavana\Desktop\ECLIPSEReader\prova.py", line 3, in
>inteHead, propertyNames, propertyTypes, propertyNumbers =
> ECLIPSEReader.init.readinspec("OPT_INJ.INSPEC")
> ValueError: failed to create intent(cache|hide)|optional array-- must
> have defined dimensions but got (-1,)
>
>
> So, I have tried with a suggestion given in the f2py mailing list, and
> I found out that this routine works:
>
>
> MODULE DUMMY
> IMPLICIT NONE
>
> ! Ok, so I want an allocatable array as output
>
> real(8), allocatable :: realOutput(:)
>
> CONTAINS
>
>subroutine AllocateDummy(dummyInput)
>
>implicit none
>save
>
>! dummyInput is *not* used, it's here just as
>! an example
>integer, intent(in) :: dummyInput
>
>! Allocate and build the output array
>allocate(realOutput(10))
>
>realOutput(1:10) = 0.0
>realOutput(3) = 3.0
>realOutput(7) = 7.0
>
>deallocate(realOutput)
>
>return
>
>end subroutine AllocateDummy
>
>
> END MODULE DUMMY
>
>
>
> But this one doesn't work:
>
>
> MODULE DUMMY
> IMPLICIT NONE
>
> ! Ok, so I want an allocatable array as output
>
> real(8), allocatable :: realOutput(:)
> integer, allocatable :: inteOutput(:)
>
> CONTAINS
>
>subroutine AllocateDummy(dummyInput)
>
>implicit none
>save
>
>! dummyInput is *not* used, it's here just as
>! an example
>integer, intent(in) :: dummyInput
>
>
>! Allocate and build the output array
>allocate(realOutput(10))
>allocate(inteOutput(20))
>
>realOutput(1:10) = 0.0
>realOutput(3) = 3.0
>realOutput(7) = 7.0
>
>inteOutput(10) = 2
>
>deallocate(realOutput)
>deallocate(inteOutput)
>
>return
>
>end subroutine AllocateDummy
>
>
> END MODULE DUMMY
>
>
>
> The difference between the 2 scripts, is just that in the second one I
> want 2 allocatable arrays instead of 1. When I compile it with f2py, I
> get this warning from getarrdims:
>
> Building modules...
>Building module "dummy"...
>Constructing F90 module support for "dummy"...
> Variables: realoutput inteoutput
> getarrdims:warning: assumed shape array, using 0 instead of ':'
> getarrdims:warning: assumed shape array, using 0 instead of ':'
>Constructing wrapper function "dummy.allocatedummy"...
> allocatedummy(dummyinput)
>
>
> Which is not present if I compile script number 1. Actually, if I run
> script 2, I can't access anymore the 2 variables realoutput and
> inteoutput (they are not there), while with script 1 I can easily
> access realoutput by writing dummy.dummy.realoutput.
> I can't actually see any big difference between the 2 scripts... am I
> missing something?
>
> This is Windows XP, Python 2.5, numpy 1.0.3.1, Compaq Visual Fortran
> 6.6, MS Visual Studio .NET 2003.
>
> Thank you for all your suggestions, I am at loss.
>
> Andrea.
>
> "Imagination Is The Only Weapon In The War Against Reality."
> http://xoomer.alice.it/infinity77/
> ___
> Numpy-discussion mailing list
> Numpy-discussion@scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
--
Lisandro D
[Numpy-discussion] [F2PY]: Allocatable Arrays
Hi All,
I sent a couple of messages to f2py mailing list, but it seems
like my problem has no simple solution so I thought to ask for some
suggestions here.
Basically, I read some huge unformatted binary files which contain
time-step data from a reservoir simulation. I don't know the
dimensions (i.e., lengths) of the vectors I am going to read, and I
find out this information only when I start reading the file. So, I
thought it would be nice to do something like:
1) Declare outputVector as allocatable;
2) Start reading the file;
3) Find the outputVector dimension and allocate it;
4) Read the data in the outputVector;
5) Return this outputVector.
It works when I compile it and build it in Fortran as an executable
(defining a "main" program in my f90 module), but it bombs when I try
to use it from Python with the error:
C:\Documents and Settings\gavana\Desktop\ECLIPSEReader>prova.py
Traceback (most recent call last):
File "C:\Documents and
Settings\gavana\Desktop\ECLIPSEReader\prova.py", line 3, in
inteHead, propertyNames, propertyTypes, propertyNumbers =
ECLIPSEReader.init.readinspec("OPT_INJ.INSPEC")
ValueError: failed to create intent(cache|hide)|optional array-- must
have defined dimensions but got (-1,)
So, I have tried with a suggestion given in the f2py mailing list, and
I found out that this routine works:
MODULE DUMMY
IMPLICIT NONE
! Ok, so I want an allocatable array as output
real(8), allocatable :: realOutput(:)
CONTAINS
subroutine AllocateDummy(dummyInput)
implicit none
save
! dummyInput is *not* used, it's here just as
! an example
integer, intent(in) :: dummyInput
! Allocate and build the output array
allocate(realOutput(10))
realOutput(1:10) = 0.0
realOutput(3) = 3.0
realOutput(7) = 7.0
deallocate(realOutput)
return
end subroutine AllocateDummy
END MODULE DUMMY
But this one doesn't work:
MODULE DUMMY
IMPLICIT NONE
! Ok, so I want an allocatable array as output
real(8), allocatable :: realOutput(:)
integer, allocatable :: inteOutput(:)
CONTAINS
subroutine AllocateDummy(dummyInput)
implicit none
save
! dummyInput is *not* used, it's here just as
! an example
integer, intent(in) :: dummyInput
! Allocate and build the output array
allocate(realOutput(10))
allocate(inteOutput(20))
realOutput(1:10) = 0.0
realOutput(3) = 3.0
realOutput(7) = 7.0
inteOutput(10) = 2
deallocate(realOutput)
deallocate(inteOutput)
return
end subroutine AllocateDummy
END MODULE DUMMY
The difference between the 2 scripts, is just that in the second one I
want 2 allocatable arrays instead of 1. When I compile it with f2py, I
get this warning from getarrdims:
Building modules...
Building module "dummy"...
Constructing F90 module support for "dummy"...
Variables: realoutput inteoutput
getarrdims:warning: assumed shape array, using 0 instead of ':'
getarrdims:warning: assumed shape array, using 0 instead of ':'
Constructing wrapper function "dummy.allocatedummy"...
allocatedummy(dummyinput)
Which is not present if I compile script number 1. Actually, if I run
script 2, I can't access anymore the 2 variables realoutput and
inteoutput (they are not there), while with script 1 I can easily
access realoutput by writing dummy.dummy.realoutput.
I can't actually see any big difference between the 2 scripts... am I
missing something?
This is Windows XP, Python 2.5, numpy 1.0.3.1, Compaq Visual Fortran
6.6, MS Visual Studio .NET 2003.
Thank you for all your suggestions, I am at loss.
Andrea.
"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] histogramdd memory needs
Hello, I use numpy.histogramdd to compute three dimensional histograms with a total number of bins in the order of 1e7. It is clear to me, that such a histogram will take a lot of memory. For a dtype=N.float64, it will take roughly 80 megabytes. However, I have the feeling that during the histogram calculation, much more memory is needed. For example, when I have data.shape = (8e6, 3) and do a numpy.histogramdd(d, 280), I expect a histogram size of (280**3)*8 = 176 megabytes, but during histogram calculation the memory need of pythonw.exe in the Windows Task Manager increases up to 687 megabytes over the level before histogram calculation. When the calculation is done, the mem usage drops down to the expected value. I assume this is due to the internal way, numpy.histogramdd works. However, when I need to calculate even bigger histograms, I cannot do it this way. So I have the following questions: 1) How can I tell histogramdd to use another dtype than float64? My bins will be very little populated so an int16 should be sufficient. Without normalization, a Integer dtype makes more sense to me. 2) Is there a way to use another algorithm (at the cost of performance) that uses less memory during calculation so that I can generate bigger histograms? My numpy version is '1.0.4.dev3937' Thanks, Lars -- Dipl.-Ing. Lars Friedrich Photonic Measurement Technology Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-Köhler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] searchsorted bug
> Try out latest SVN. It should have this problem fixed. Thanks for this. I've realized that for my case, using object arrays is probably best. I still think that long term it would be good to allow comparison functions to take different types, so that one could compare say integer arrays with floating point arrays without doing an upcast. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
