Dear Craig and Chris,
thank you for suggestions.
I checked the following with a linux system 8Gb in total with padded
numbers. The data I have to process are less than 4194304 elements, so I
m fine.
use PDL::FFT
$d=random(4194304)
p $d->info('%M') # 32.00MB
realfft($d) # => OK
#--- next step
$d=random(8388608)
p $d->info('%M') # 64.00MB
realfft($d)
Error: fftradix() - exceeded number of factors
Error: fftradix() - insufficient memory.
best regards
Frank
On 13.02.2013 23:00, [email protected] wrote:
Send Perldl mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Perldl digest..."
Today's Topics:
1. Re: PDL::FFT maximum number of elements to process
(Craig DeForest)
2. Re: PDL::FFT maximum number of elements to process
(Chris Marshall)
----------------------------------------------------------------------
Message: 1
Date: Wed, 13 Feb 2013 08:27:02 -0700
From: Craig DeForest<[email protected]>
To: Chris Marshall<[email protected]>
Cc: [email protected], Frank Boers<[email protected]>
Subject: Re: [Perldl] PDL::FFT maximum number of elements to process
Message-ID:<[email protected]>
Content-Type: text/plain; charset=us-ascii
Sorry I didn't catch this earlier. On a 64-bit system it's really just your
available memory on your particular system. You need enough free memory to
hold a bit over 2x the storage space for the array itself (as a complex number,
so 4x if you're transforming a real array), after you round the dimensions up
to the nearest power of two. I've done a fair number of 1024x1024x256
floating-point 3-D FFTs with fftnd. For some reason, fftnd requires the full
FFT buffer size for each dimension (that could/should be fixed, but it's easier
to buy RAM), so that is 12 GiB (0.25GiB x 4 x 2 x 2 x 3).
If you're close to your system RAM size you could run into fragmentation issues
since the FFT buffers have to be contiguous in the address space and you've
presumably been mallocking and freeing hunks of memory all over the place. In
that case you can push the variable out to storage (e.g. as a FITS file), then
Fourier transform it with a standalone script and slurp up the result in your
main script. That lets the standalone script have its own address space
independent of the original one. But that's just a workaround for the main
issue, which is that you need a ton of RAM to transform large arrays, even with
the FFT.
On Feb 13, 2013, at 5:09 AM, Chris Marshall<[email protected]> wrote:
I don't know the maximum but if you
pick something with a smaller number
of prime factors, you should be able
to do a larger FFT (within your memory
and pdl size limits).
You could try PDL::FFTW but you
will need to install version 2 of the
FFTW library.
--Chris
On Wed, Feb 13, 2013 at 3:35 AM, Frank Boers<[email protected]> wrote:
Dear PDLs,
what is the maximum number of elements PDL::FFT can process?
use PDL::FFT
$data=random(8000000)
realfft($data)
$data=random(8100000)
realfft($data)
Error: fftradix() - exceeded number of factors
Error: fftradix() - insufficient memory.
best regards
Frank
--
Frank Boers
Institute of Neuroscience and Medicine - 4
Medical Imaging Physics
Forschungszentrum Juelich GmbH
52425 Juelich
phone: +49 - (0)2461-61-6005
fax : +49 - (0)2461-61-2820
email: [email protected]
http://www.fz-juelich.de/inm/inm-4/DE/Forschung/MEG-Physik/_node.html
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
------------------------------
Message: 2
Date: Wed, 13 Feb 2013 10:53:45 -0500
From: Chris Marshall<[email protected]>
To: Craig DeForest<[email protected]>
Cc: [email protected], Frank Boers<[email protected]>
Subject: Re: [Perldl] PDL::FFT maximum number of elements to process
Message-ID:
<CAPTtexKQUKaQ3qGiUKaijE=bmWXoap=qq1zd5rdrme2ob3s...@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
<shameless plug for PDL development volunteers>
I would like to point out that FFTW3 has *extremely*
general FFT capabilities, supports multiple threads,
parallelization, in-place operation, as well as a much
more flexible set of calling schemes.
Porting a PDL::FFTW3 interface would make all of
this possible. :-)
--Chris
On Wed, Feb 13, 2013 at 10:27 AM, Craig DeForest
<[email protected]> wrote:
Sorry I didn't catch this earlier. On a 64-bit system it's really just your
available memory on your particular system. You need enough free memory to
hold a bit over 2x the storage space for the array itself (as a complex number,
so 4x if you're transforming a real array), after you round the dimensions up
to the nearest power of two. I've done a fair number of 1024x1024x256
floating-point 3-D FFTs with fftnd. For some reason, fftnd requires the full
FFT buffer size for each dimension (that could/should be fixed, but it's easier
to buy RAM), so that is 12 GiB (0.25GiB x 4 x 2 x 2 x 3).
If you're close to your system RAM size you could run into fragmentation issues
since the FFT buffers have to be contiguous in the address space and you've
presumably been mallocking and freeing hunks of memory all over the place. In
that case you can push the variable out to storage (e.g. as a FITS file), then
Fourier transform it with a standalone script and slurp up the result in your
main script. That lets the standalone script have its own address space
independent of the original one. But that's just a workaround for the main
issue, which is that you need a ton of RAM to transform large arrays, even with
the FFT.
On Feb 13, 2013, at 5:09 AM, Chris Marshall<[email protected]> wrote:
I don't know the maximum but if you
pick something with a smaller number
of prime factors, you should be able
to do a larger FFT (within your memory
and pdl size limits).
You could try PDL::FFTW but you
will need to install version 2 of the
FFTW library.
--Chris
On Wed, Feb 13, 2013 at 3:35 AM, Frank Boers<[email protected]> wrote:
Dear PDLs,
what is the maximum number of elements PDL::FFT can process?
use PDL::FFT
$data=random(8000000)
realfft($data)
$data=random(8100000)
realfft($data)
Error: fftradix() - exceeded number of factors
Error: fftradix() - insufficient memory.
best regards
Frank
--
Frank Boers
Institute of Neuroscience and Medicine - 4
Medical Imaging Physics
Forschungszentrum Juelich GmbH
52425 Juelich
phone: +49 - (0)2461-61-6005
fax : +49 - (0)2461-61-2820
email: [email protected]
http://www.fz-juelich.de/inm/inm-4/DE/Forschung/MEG-Physik/_node.html
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
------------------------------
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
End of Perldl Digest, Vol 93, Issue 20
**************************************
--
Frank Boers
Institute of Neuroscience and Medicine - 4
Medical Imaging Physics
Forschungszentrum Juelich GmbH
52425 Juelich
phone: +49 - (0)2461-61-6005
fax : +49 - (0)2461-61-2820
email: [email protected]
http://www.fz-juelich.de/inm/inm-4/DE/Forschung/MEG-Physik/_node.html
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
