Re: [Numpy-discussion] fourier with single precision
Hello, thanks for your comments. If I got you right, I should look for a FFT-code that uses SSE (what does this actually stand for?), which means that it vectorizes 32bit-single-operations into larger chunks that make efficient use of recent CPUs. You mentioned FFTW and MKL. Is this www.fftw.org and the 'intel math kernel library'? If I would like to use one of them, is numpy the right place to put it in? Does anyone know, if it is possible to switch on SSE support (at compile time) in the fftpack.c that numpy uses? Thanks Lars ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multinomial error?
On Fri, 03 Aug 2007, [EMAIL PROTECTED] apparently wrote: > I appear to be having a problem with the random.multinomial function. For > some > reason if i attempt to loop over a large number of single-trial multinomial > picks then the function begins to ignore some non-zero entries in my 1-D > array > of multinomial probabilities... Is seems that there is no upper limit on the > size of the probability array for a one off multinomial pick, but if looping > over the process multiple times the function can't handle the whole array and > seems to truncate it arbitrarily before performing the trial with only the > remaining probabilities. Minimal example? Cheers, Alan Isaac ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy arrays, data allocation and SIMD alignement
On Aug 4, 3:24 am, "Anne Archibald" <[EMAIL PROTECTED]> wrote: > It seems to me two things are needed: > > * A mechanism for requesting numpy arrays with buffers aligned to an > arbitrary power-of-two size (basically just using posix_memalign or > some horrible hack on platforms that don't have it). Right, you might as well allow the alignment (to a power-of-two size) to be specified at runtime, as there is really no cost to implementing an arbitrary alignment once you have any alignment. Although you should definitely use posix_memalign (or the old memalign) where it is available, unfortunately it's not implemented on all systems. e.g. MacOS X and FreeBSD don't have it, last I checked (although in both cases their malloc is 16-byte aligned). Microsoft VC ++ has a function called _aligned_malloc which is equivalent. However, since MinGW (www.mingw.org) didn't have an _aligned_malloc function, I wrote one for them a few years ago and put it in the public domain (I use MinGW to cross-compile to Windows from Linux and need the alignment). You are free to use it as a fallback on systems that don't have a memalign function if you want. It should work on any system where sizeof(void*) is a power of two (i.e. every extant architecture, that I know of). You can download it and its test program from: ab-initio.mit.edu/~stevenj/align.c ab-initio.mit.edu/~stevenj/tstalign.c It just uses malloc with a little extra padding as needed to align the data, plus a copy of the original pointer so that you can still free and realloc (using _aligned_free and _aligned_realloc). It could be made a bit more efficient, but it probably doesn't matter. > * A macro (in C, and some way to get the same information from python, > perhaps just "a.ctypes.data % 16") to test for common alignment cases; > SIMD alignment and arbitrary power-of-two alignment are probably > sufficient. In C this is easy, just ((uintptr_t) pointer) % 16 == 0. You might also consider a way to set the default alignment of numpy arrays at runtime, rather than requesting aligned arrays individually. e.g. so that someone could come along at a later date to a large program and just add one function call to make all the arrays 16-byte aligned to improve performance using SIMD libraries. Regards, Steven G. Johnson ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] multinomial error?
Hi, I appear to be having a problem with the random.multinomial function. For some reason if i attempt to loop over a large number of single-trial multinomial picks then the function begins to ignore some non-zero entries in my 1-D array of multinomial probabilities... Is seems that there is no upper limit on the size of the probability array for a one off multinomial pick, but if looping over the process multiple times the function can't handle the whole array and seems to truncate it arbitrarily before performing the trial with only the remaining probabilities. There is a reason why i need to loop over a large number of single-trial events, rather than just replacing the loop with a large number of trials in one single multinomial pick (annoying, as that's so much quicker!). Thanks for any help, Adam ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How to implement a 'pivot table?'
Generating these types of summary statistics is very common in SAS. In SAS you would set up a sequence of procedures. First sort by the variables of interest and then calculate the metrics of interest by the combination of values. In numpy/scipy this might be something like: 1. Sort by date and region 2. Determine 1st and last index of the blocks 3. Calculate mean,sum, etc. for each of the blocks. Based on the earlier arguments I am wondering, however, if this would provide any speed up. I am very interested in this issue so if you implement some general procedures and perhaps speed tests please share them with the list. Best, Vincent On Jul 31, 10:00 am, "Geoffrey Zhu" <[EMAIL PROTECTED]> wrote: > Hi Timothy, > > On 7/31/07, Timothy Hochberg <[EMAIL PROTECTED]> wrote: > > > [SNIP] > > > > The 'brute-force' way is basically what you suggested -- looping > > > through all the records and building a two-way hash-table of the data. > > > > The problem of the brute-force' approach is that it is not taking > > > advantage of facilities of numpy and can be slow in speed. If only > > > there is some built-in mechanism in numpy to handle this. > > > I'm curious; have you tried this and found it slow, or is this a hunch based > > on the reputed slowness of Python? Algorithmically, you can't do much > > better: the dictionary and set operations are O(1), so the whole operation > > is O(N), and you won't do any better than that, order wise. What your left > > with is trying to reduce constant factors. > > I agree that algorithmically you can't do much better. It is basically > a C vs Python thing. One advantage of numpy is that you can do > vectorized operations at the speed of C. With this algorithm, we have > to process the data element by element and the speed advantage of > numpy is lost. Since data has to be stored in python sets and maps, I > imagine the storage advantage is also lost. > > I was in fact looking for some implemention of this algorithm in numpy > (and so C) that does exactly this, or some implementation of this > algorithm that can leverage the fast numpy routines to do this. > > I haven't tried it with the real data load yet. I know the number of > records will be huge and it is just a hunch that it will be slow. > > > There are various ways one might go about reducing constant factors, but > > they depend on the details of the problem. For example, if the dates are > > dense and you are going to parse them anyway, you could replace the hash > > with table that you index into with the date as an integer. I'm not sure > > that you are going to do a lot better than the brute force algorithm in the > > generic force case though. > > Unfortunately it has to be something generic. > > Thanks a lot for your help. > Geoffrey > ___ > Numpy-discussion mailing list > [EMAIL PROTECTED]://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy installation problem
Hi, I downloaded numpy1.0.3-2.tar and unzipped and untared. However somehow new numpy does not work. It invokes the old numpy 0.9.6 when i import numpy from python and type in numpy.version.version . I tried to change path and once I do that and when I do import numpy it says "running from source directory" and then if I try numpy.version.version it gives some error. Is there something obvious I am missing after unzipping and untaring the numpy source file ? For example do I need to do something to install the new numpy1.0.3 ? Or do I also need to download full python package ? I am trying to run this on Red Hat Linux 3.2.2-5 which has a gcc 3.2.2 and the version of python is 2.4 . Any help will be greatly appreciated. Cheers Kings ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
