Looking at the source, the maximum number of factors is 11 and your number is 2**23 or 4**11.5 which is why the fail. You can either: - rebuild PDL with a different value for NFACTORS - pick a number that has fewer factors - install FFTW2 and rebuild PDL to use it
For example, if I use a size of 5**10 then the FFT completes with the existing code. I don't know about speed tradeoffs. Cheers, Chris On Thu, Feb 14, 2013 at 10:22 AM, Frank Boers <[email protected]> wrote: > Dear Craig and Chris, > thank you for suggestions. > I checked the following with a linux system 8Gb in total with padded > numbers. The data I have to process are less than 4194304 elements, so I m > fine. > > > use PDL::FFT > > $d=random(4194304) > p $d->info('%M') # 32.00MB > > realfft($d) # => OK > > #--- next step > $d=random(8388608) > p $d->info('%M') # 64.00MB > > realfft($d) > > Error: fftradix() - exceeded number of factors > Error: fftradix() - insufficient memory. > > best regards > Frank > > > > On 13.02.2013 23:00, [email protected] wrote: >> >> Send Perldl mailing list submissions to >> [email protected] >> >> To subscribe or unsubscribe via the World Wide Web, visit >> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl >> or, via email, send a message with subject or body 'help' to >> [email protected] >> >> You can reach the person managing the list at >> [email protected] >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of Perldl digest..." >> >> >> Today's Topics: >> >> 1. Re: PDL::FFT maximum number of elements to process >> (Craig DeForest) >> 2. Re: PDL::FFT maximum number of elements to process >> (Chris Marshall) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Wed, 13 Feb 2013 08:27:02 -0700 >> From: Craig DeForest<[email protected]> >> To: Chris Marshall<[email protected]> >> Cc: [email protected], Frank Boers<[email protected]> >> Subject: Re: [Perldl] PDL::FFT maximum number of elements to process >> Message-ID:<[email protected]> >> Content-Type: text/plain; charset=us-ascii >> >> >> Sorry I didn't catch this earlier. On a 64-bit system it's really just >> your available memory on your particular system. You need enough free >> memory to hold a bit over 2x the storage space for the array itself (as a >> complex number, so 4x if you're transforming a real array), after you round >> the dimensions up to the nearest power of two. I've done a fair number of >> 1024x1024x256 floating-point 3-D FFTs with fftnd. For some reason, fftnd >> requires the full FFT buffer size for each dimension (that could/should be >> fixed, but it's easier to buy RAM), so that is 12 GiB (0.25GiB x 4 x 2 x 2 x >> 3). >> >> If you're close to your system RAM size you could run into fragmentation >> issues since the FFT buffers have to be contiguous in the address space and >> you've presumably been mallocking and freeing hunks of memory all over the >> place. In that case you can push the variable out to storage (e.g. as a >> FITS file), then Fourier transform it with a standalone script and slurp up >> the result in your main script. That lets the standalone script have its >> own address space independent of the original one. But that's just a >> workaround for the main issue, which is that you need a ton of RAM to >> transform large arrays, even with the FFT. >> >> >> >> On Feb 13, 2013, at 5:09 AM, Chris Marshall<[email protected]> >> wrote: >> >>> I don't know the maximum but if you >>> pick something with a smaller number >>> of prime factors, you should be able >>> to do a larger FFT (within your memory >>> and pdl size limits). >>> >>> You could try PDL::FFTW but you >>> will need to install version 2 of the >>> FFTW library. >>> >>> --Chris >>> >>> On Wed, Feb 13, 2013 at 3:35 AM, Frank Boers<[email protected]> >>> wrote: >>>> >>>> Dear PDLs, >>>> what is the maximum number of elements PDL::FFT can process? >>>> >>>> use PDL::FFT >>>> >>>> $data=random(8000000) >>>> >>>> realfft($data) >>>> >>>> $data=random(8100000) >>>> >>>> realfft($data) >>>> Error: fftradix() - exceeded number of factors >>>> Error: fftradix() - insufficient memory. >>>> >>>> >>>> best regards >>>> Frank >>>> >>>> -- >>>> Frank Boers >>>> Institute of Neuroscience and Medicine - 4 >>>> Medical Imaging Physics >>>> Forschungszentrum Juelich GmbH >>>> 52425 Juelich >>>> phone: +49 - (0)2461-61-6005 >>>> fax : +49 - (0)2461-61-2820 >>>> email: [email protected] >>>> http://www.fz-juelich.de/inm/inm-4/DE/Forschung/MEG-Physik/_node.html >>>> >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------------------------ >>>> >>>> ------------------------------------------------------------------------------------------------ >>>> Forschungszentrum Juelich GmbH >>>> 52425 Juelich >>>> Sitz der Gesellschaft: Juelich >>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher >>>> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), >>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, >>>> Prof. Dr. Sebastian M. Schmidt >>>> >>>> ------------------------------------------------------------------------------------------------ >>>> >>>> ------------------------------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> Perldl mailing list >>>> [email protected] >>>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl >>> >>> >>> _______________________________________________ >>> Perldl mailing list >>> [email protected] >>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl >>> >> >> >> >> >> ------------------------------ >> >> Message: 2 >> Date: Wed, 13 Feb 2013 10:53:45 -0500 >> From: Chris Marshall<[email protected]> >> To: Craig DeForest<[email protected]> >> Cc: [email protected], Frank Boers<[email protected]> >> Subject: Re: [Perldl] PDL::FFT maximum number of elements to process >> Message-ID: >> >> <CAPTtexKQUKaQ3qGiUKaijE=bmWXoap=qq1zd5rdrme2ob3s...@mail.gmail.com> >> Content-Type: text/plain; charset=ISO-8859-1 >> >> >> <shameless plug for PDL development volunteers> >> >> I would like to point out that FFTW3 has *extremely* >> general FFT capabilities, supports multiple threads, >> parallelization, in-place operation, as well as a much >> more flexible set of calling schemes. >> >> Porting a PDL::FFTW3 interface would make all of >> this possible. :-) >> >> --Chris >> >> On Wed, Feb 13, 2013 at 10:27 AM, Craig DeForest >> <[email protected]> wrote: >>> >>> Sorry I didn't catch this earlier. On a 64-bit system it's really just >>> your available memory on your particular system. You need enough free >>> memory to hold a bit over 2x the storage space for the array itself (as a >>> complex number, so 4x if you're transforming a real array), after you round >>> the dimensions up to the nearest power of two. I've done a fair number of >>> 1024x1024x256 floating-point 3-D FFTs with fftnd. For some reason, fftnd >>> requires the full FFT buffer size for each dimension (that could/should be >>> fixed, but it's easier to buy RAM), so that is 12 GiB (0.25GiB x 4 x 2 x 2 x >>> 3). >>> >>> If you're close to your system RAM size you could run into fragmentation >>> issues since the FFT buffers have to be contiguous in the address space and >>> you've presumably been mallocking and freeing hunks of memory all over the >>> place. In that case you can push the variable out to storage (e.g. as a >>> FITS file), then Fourier transform it with a standalone script and slurp up >>> the result in your main script. That lets the standalone script have its >>> own address space independent of the original one. But that's just a >>> workaround for the main issue, which is that you need a ton of RAM to >>> transform large arrays, even with the FFT. >>> >>> >>> >>> On Feb 13, 2013, at 5:09 AM, Chris Marshall<[email protected]> >>> wrote: >>> >>>> I don't know the maximum but if you >>>> pick something with a smaller number >>>> of prime factors, you should be able >>>> to do a larger FFT (within your memory >>>> and pdl size limits). >>>> >>>> You could try PDL::FFTW but you >>>> will need to install version 2 of the >>>> FFTW library. >>>> >>>> --Chris >>>> >>>> On Wed, Feb 13, 2013 at 3:35 AM, Frank Boers<[email protected]> >>>> wrote: >>>>> >>>>> Dear PDLs, >>>>> what is the maximum number of elements PDL::FFT can process? >>>>> >>>>> use PDL::FFT >>>>> >>>>> $data=random(8000000) >>>>> >>>>> realfft($data) >>>>> >>>>> $data=random(8100000) >>>>> >>>>> realfft($data) >>>>> Error: fftradix() - exceeded number of factors >>>>> Error: fftradix() - insufficient memory. >>>>> >>>>> >>>>> best regards >>>>> Frank >>>>> >>>>> -- >>>>> Frank Boers >>>>> Institute of Neuroscience and Medicine - 4 >>>>> Medical Imaging Physics >>>>> Forschungszentrum Juelich GmbH >>>>> 52425 Juelich >>>>> phone: +49 - (0)2461-61-6005 >>>>> fax : +49 - (0)2461-61-2820 >>>>> email: [email protected] >>>>> http://www.fz-juelich.de/inm/inm-4/DE/Forschung/MEG-Physik/_node.html >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------------------------ >>>>> >>>>> ------------------------------------------------------------------------------------------------ >>>>> Forschungszentrum Juelich GmbH >>>>> 52425 Juelich >>>>> Sitz der Gesellschaft: Juelich >>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher >>>>> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), >>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, >>>>> Prof. Dr. Sebastian M. Schmidt >>>>> >>>>> ------------------------------------------------------------------------------------------------ >>>>> >>>>> ------------------------------------------------------------------------------------------------ >>>>> >>>>> _______________________________________________ >>>>> Perldl mailing list >>>>> [email protected] >>>>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl >>>> >>>> >>>> _______________________________________________ >>>> Perldl mailing list >>>> [email protected] >>>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl >>>> >>> >> >> >> >> ------------------------------ >> >> _______________________________________________ >> Perldl mailing list >> [email protected] >> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl >> >> >> End of Perldl Digest, Vol 93, Issue 20 >> ************************************** > > > -- > Frank Boers > Institute of Neuroscience and Medicine - 4 > Medical Imaging Physics > Forschungszentrum Juelich GmbH > 52425 Juelich > phone: +49 - (0)2461-61-6005 > fax : +49 - (0)2461-61-2820 > email: [email protected] > http://www.fz-juelich.de/inm/inm-4/DE/Forschung/MEG-Physik/_node.html > > > _______________________________________________ > Perldl mailing list > [email protected] > http://mailman.jach.hawaii.edu/mailman/listinfo/perldl _______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
