[Numpy-discussion] Anyone coming to the sprints have a >= 1GB memory stick or blank CD?

[David Goldsmith]

If so, and I could use it to try to install Kubuntu tomorrow, I'd really 
appreciate it if you'd bring it w/ you.  Thanks!

DG

__
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fwd: GPU Numpy

[Sturla Molden]

Erik Tollerud skrev:
>> NumPy arrays on the GPU memory is an easy task. But then I would have to
>> write the computation in OpenCL's dialect of C99? 
> This is true to some extent, but also probably difficult to do given
> the fact that paralellizable algorithms are generally more difficult
> to formulate in striaghtforward ways. 
Then you have misunderstood me completely. Creating an ndarray that has 
a buffer in graphics memory is not too difficult, given that graphics 
memory can be memory mapped. This has nothing to do with parallelizable 
algorithms or not. It is just memory management. We could make an 
ndarray subclass that quickly puts is content in a buffer accessible to 
the GPU. That is not difficult. But then comes the question of what you 
do with it.

I think many here misunderstands the issue here:

Teraflops peak performance of modern GPUs is impressive. But NumPy 
cannot easily benefit from that. In fact, there is little or nothing to 
gain from optimising in that end. In order for a GPU to help, 
computation must be the time-limiting factor. It is not. There is not 
more to say about using GPUs in NumPy right now.

Take a look at the timings here: http://www.scipy.org/PerformancePython 
It shows that computing with NumPy is more than ten times slower than 
using plain C. This is despite NumPy being written in C. The NumPy code 
does not incur 10 times more floating point operations than the C code. 
The floating point unit does not run in turtle mode when using NumPy. 
NumPy's relative slowness compared to C has nothing to do with floating 
point computation. It is due to inferior memory use (temporary buffers, 
multiple buffer traversals) and memory access being slow. Moving 
computation to the GPU can only make this worse.

Improved memory usage - e.g. through lazy evaluation and JIT compilaton 
of expressions - can give up to a tenfold increase in performance. That 
is where we must start optimising to get a faster NumPy. Incidentally, 
this will  also make it easier to leverage on modern GPUs.

Sturla Molden
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GPU Numpy

[Nicolas Pinto]

Agreed! What would be the best name? Our package will provide non-pythonic
bindings to cuda (e.g. import cuda; cuda.cudaMemcpy( ... ) ) and some numpy
sugar (e.g. from cuda import sugar; sugar.fft.fftconvolve(ndarray_a,
ndarray_b, 'same')).

How about cuda-ctypes or ctypes-cuda? Any suggestion?

At the same time we may wait for Nvidia to unlock this Driver/Runtime issue,
so we don't need this anymore.

Best,

N

2009/8/21 Stéfan van der Walt 

> 2009/8/21 Nicolas Pinto :
> > For those of you who are interested, we forked python-cuda recently and
> > started to add some numpy "sugar". The goal of python-cuda is to
> > *complement* PyCuda by providing an equivalent to the CUDA Runtime API
> > (understand: not Pythonic) using automatically-generated ctypes bindings.
> > With it you can use CUBLAS, CUFFT and the emulation mode (so you don't
> need
> > a GPU to develop):
> > http://github.com/npinto/python-cuda/tree/master
>
> Since you forked the project, it may be worth giving it a new name.
> PyCuda vs. python-cuda is bound to confuse people horribly!
>
> Cheers
> Stéfan
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Nicolas Pinto
Ph.D. Candidate, Brain & Computer Sciences
Massachusetts Institute of Technology, USA
http://web.mit.edu/pinto
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GPU Numpy

[Stéfan van der Walt]

2009/8/21 Nicolas Pinto :
> For those of you who are interested, we forked python-cuda recently and
> started to add some numpy "sugar". The goal of python-cuda is to
> *complement* PyCuda by providing an equivalent to the CUDA Runtime API
> (understand: not Pythonic) using automatically-generated ctypes bindings.
> With it you can use CUBLAS, CUFFT and the emulation mode (so you don't need
> a GPU to develop):
> http://github.com/npinto/python-cuda/tree/master

Since you forked the project, it may be worth giving it a new name.
PyCuda vs. python-cuda is bound to confuse people horribly!

Cheers
Stéfan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GPU Numpy

[Nicolas Pinto]

Hello

from gpunumpy import *
> x=zeros(100,dtype='gpufloat') # Creates an array of 100 elements on the GPU
> y=ones(100,dtype='gpufloat')
> z=exp(2*x+y) # z in on the GPU, all operations on GPU with no transfer
> z_cpu=array(z,dtype='float') # z is copied to the CPU
> i=(z>2.3).nonzero()[0] # operation on GPU, returns a CPU integer array
>

PyCuda already supports this through the gpuarray interface. As soon as
Nvidia allows us to combine Driver and Runtime APIs, we'll be able to
integrate libraries like CUBLAS, CUFFT, and any other runtime-depedent
library. We could probably get access to CUBLAS/CUFFT source code as Nvidia
released the 1.1 version in the past:
http://sites.google.com/site/cudaiap2009/materials-1/extras/online-resources#TOC-CUBLAS-and-CUFFT-1.1-Source-Code
but it would be easier to just use the libraries (and 1.1 is outdated now).

For those of you who are interested, we forked python-cuda recently and
started to add some numpy "sugar". The goal of python-cuda is to
*complement* PyCuda by providing an equivalent to the CUDA Runtime API
(understand: not Pythonic) using automatically-generated ctypes bindings.
With it you can use CUBLAS, CUFFT and the emulation mode (so you don't need
a GPU to develop):
http://github.com/npinto/python-cuda/tree/master

HTH

Best,

-- 
Nicolas Pinto
Ph.D. Candidate, Brain & Computer Sciences
Massachusetts Institute of Technology, USA
http://web.mit.edu/pinto
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerating NumPy computations [Was: GPU Numpy]

[Paul Ivanov]

Matthew Brett, on 2009-08-21 11:51,  wrote:
> Hi,
> 
> > Indeed. In the future, if OpenCL is the way to go, it may even be
> > helpful to have Numpy using OpenCL directly, as AMD provides an SDK
> > for OpenCL, and with Larrabee approaching, Intel will surely provide
> > one of its own.
> 
> I was just in a lecture by one of the Intel people about OpenCL:
> 
> http://parlab.eecs.berkeley.edu/bootcampagenda
> http://parlab.eecs.berkeley.edu/sites/all/parlab/files/OpenCL_Mattson.pdf
> 
> He offered no schedule for an Intel OpenCL implementation, but said
> that they were committed to it.
> 
> The lectures in general were effective in pointing out what a
> time-consuming effort it can be moving algorithms into the the
> parallel world - including GPUs.  The lecture just passed cited the
> example of a CUDA-based BLAS implementation on the GPU that was slower
> than the CPU version.Making BLAS go faster required a lot of work
> to find optimal strategies for blocking, transfer between CPU / GPU
> shared memory / GPU registers, vector sizes and so on - this on a
> specific NVIDIA architecture.
> 
> I can imagine Numpy being useful for scripting in this
> C-and-assembler-centric world, making it easier to write automated
> testers, or even generate C code.
> 
> Is anyone out there working on this kind of stuff?  I ask only because
> there seems to be considerable interest here on the Berkeley campus.

This is exactly the sort of thing you can do with PyCUDA, which makes it
so awesome!  


In particular, see the metaprogramming portion of the docs:


The metaprogramming section of the slides and source code from Nicolas
Pinto and Andreas Klöckner *excellent* SciPy2009 Tutorials is even more 
thorough:




cheers,
Paul Ivanov

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PRNGs and multi-threading

[Sturla Molden]

Xavier Saint-Mleux skrev:
> Of course, the mathematically correct way would be to use a correct
> jumpahead function, but all the implementations that I know of are GPL. 
> A recent article about this is:
>
> www.iro.umontreal.ca/~lecuyer/myftp/papers/jumpmt.pdf
>
>   
I know of no efficient "jumpahead" function for MT. Several seconds for 
1000 jumps ahead is not impressive -- just generating the deviates is 
faster!

With DCMT it is easy to create "independent" MTs with smaller periods. 
Independence here means that the "characteristic polynomials are 
relatively prime to each other". A "small" period of e.g. 2**521 - 1 
means that if we produce 1 billion deviates per minute, it would still 
take the MT about 10**143 years to cycle. Chances are we will not be 
around to see that happen. It also seems that nvidia has endorsed this 
method:

http://developer.download.nvidia.com/compute/cuda/sdk/website/projects/MersenneTwister/doc/MersenneTwister.pdf



S.M.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problems distributing NumPy

[Michael Cooper]

Hi again-

 

I've been working on this problem off and on over the last couple of years,
and of course did not find the solution until I finally broke down and
posted to the list. It looks like the problem is very simple: Although I was
adding my private folder to the Python path, I was not doing so until after
NumPy was initialized. Hence, the library could not be found. It is, in
fact, as simple as I had hoped it would be to distribute NumPy in a private
folder.

 

Thanks,

 

Michael

 

  _  

From: numpy-discussion-boun...@scipy.org
[mailto:numpy-discussion-boun...@scipy.org] On Behalf Of Michael Cooper
Sent: August 21, 2009 12:35 PM
To: numpy-discussion@scipy.org
Subject: [Numpy-discussion] Problems distributing NumPy

 

Hi all-

 

I am writing a C++ application with embedded Python scripting. Some of the
scripts use NumPy, so I have been working out the best way to distribute
NumPy with my software. At the moment, I've got a private folder which I add
to the Python path. In this folder, I include all the files which would
usually get installed to the "site-packages" folder. To get these files,
I've simply unzipped the distutils installer (for NumPy, I am using the "no
SSE" version at the moment), and copied the contents of the resulting
"PLATLIB" folder to my private folder.

 

For all the other libraries I am using, this method seems to work fine.
However, with NumPy, if I do things this way, when I call "import_array()",
it jumps back out of the calling function, skipping the rest of that
function, and continues from there. If I install NumPy in the usual way,
this does not happen. It seems like the NumPy initialization is failing when
I install into the private folder, but not if I use the normal installer.

 

Many of the users of my software aren't particularly Python savvy, so having
them install everything manually is not an option. I would like to avoid
having my own installer call external installers, sine that's confusing for
some users. Finally, if possible, I would like to avoid changing the user's
"Python26" folder, since I have no way of knowing what else might be relying
on its contents. Lots of searching on how to install third-party libraries
has led me to the "PLATLIB" method, so I'm at a bit of a loss as to what
else to try. Does anyone here know what might be going wrong?

 

I am using Python 2.6, Boost 1.38, and NumPy 1.3.0 on a Windows XP system.
The embedding program is written in C++, and compiled using Visual Studio
2005.

 

Thanks,

 

Michael

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.61/2313 - Release Date: 08/21/09
06:04:00


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerating NumPy computations [Was: GPU Numpy]

[James Bergstra]

On Fri, Aug 21, 2009 at 2:51 PM, Matthew Brett wrote:
> I can imagine Numpy being useful for scripting in this
> C-and-assembler-centric world, making it easier to write automated
> testers, or even generate C code.
>
> Is anyone out there working on this kind of stuff?  I ask only because
> there seems to be considerable interest here on the Berkeley campus.
>
> Best,
>
> Matthew

Frederic Bastien and I are working on this sort of thing.  We use a
project called theano to build symbolic expression graphs.  Theano
optimizes those graphs like an optimizing compiler, and then it
generates C code for those graphs.  We haven't put a lot of effort
into optimizing the C implementations of most expressions (except for
non-separable convolution), but we call fast blas and fftw functions,
and our naive implementations are typically faster than equivalent
numpy expressions just because they are in C.  (Although congrats to
those working at optimizing numpy... it has gotten a lot faster over
the last few years!)

We are now writing another backend that generates cuda runtime C++.
It is just like you say: even for simple tasks like adding two vectors
together or summing the elements of a matrix, there are several
possible kernels that can be optimal in different circumstances.  The
penalty of choosing a sub-optimal kernel can be pretty high.  So what
ends up happening is that even for simple ufunc-type expressions, we
have
- a version for when the arguments are small and everything is c-contiguous
- a general version that is typically orders of magnitude slower than
the optimal choice
- versions for when arguments are small and 1D, 2D, 3D, 4D, 5D
- versions for when various of the arguments are broadcasted in different ways
- versions for when there is at least one large contiguous dimension

And the list goes on.  We are still in the process of understanding
the architecture and the most effective strategies for optimization.
I think our design is a good one though from the users' perspective
because it supports a completely opaque front-end.. you just program
the symbolic graph in python using normal expressions, compile it as a
function, and call it.   The detail of whether it is evaluated on the
CPU or the GPU (or both) is hidden.

If anyone is interested in what we're doing please feel free to send
me an email.  Links to these projects are

http://www.pylearn.org/theano
http://code.google.com/p/theano-cuda-ndarray/
http://code.google.com/p/cuda-ndarray/

James
-- 
http://www-etud.iro.umontreal.ca/~bergstrj
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PRNGs and multi-threading

[Xavier Saint-Mleux]

Robert Kern wrote:
> On Fri, Aug 21, 2009 at 20:50, Sturla Molden wrote:
>   
>> Sturla Molden skrev:
>> 
>>> It seems there is a special version of the Mersenne Twister for this.
>>> The code is LGPL (annoying for SciPy but ok for me).
>>>   
>> http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/DC/dgene.pdf
>> 
>>
>> Basically it encodes the thread-ids in the characteristic polynomial of
>> the MT, producing multiple small-period, independent MTs. That solves it
>> then. Too bad this is LGPL. It would be a very useful enhancement to
>> RandomState.
>> 
>
> I agree. It might be possible to re-implement it from the original
> papers, but it's a chunk of work.
>
>   
I use the following PRNG class, derived from RandomState, which allows a
PRNG to create multiple different sub-PRNGs in a deterministic way:

http://bazaar.launchpad.net/~piaget-dev/piaget/dev/annotate/head%3A/piaget/math/prng.py

It is written in Python and has an Apache license (BSD-like).  There is
no mathematical proof that different PRNGs will have states "far enough"
from each other, but it works well in practice (I've had bad surprises
using Python's random.jumpahead, which is not a real jumpahead).

Of course, the mathematically correct way would be to use a correct
jumpahead function, but all the implementations that I know of are GPL. 
A recent article about this is:

www.iro.umontreal.ca/~lecuyer/myftp/papers/jumpmt.pdf


Xavier Saint-Mleux



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] A better median function?

[Matthew Brett]

> On Fri, Aug 21, 2009 at 11:33 AM, Matthew Brett 
> wrote:
>> Nicolas investigated algorithms that find the lower (or upper) median
>> value.  The lower median is the median iff there are an odd number of
>> entries in our list, or the lower of the central values in the sort,
>> when there are an even number of values in the list. So, we need the
>> upper _and_ lower median when there are an even number of entries.  I
>> guess the necessity of doing those two related searches may change the
>> relative strengths of the algorithms, but I'm sure this is a
>> well-known problem.
>
> My trivial solution to this was that since the data is now in two
> "partitions", all one needs to do is quickly scan through the upper
> partition to find the minimum value.
...
> Brain dead, perhaps, but it worked for my test.

Nice...  Your brain, when dead, is working better than mine,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] A better median function?

[Mike Ressler]

Hi,

On Fri, Aug 21, 2009 at 11:33 AM, Matthew Brett wrote:
> Nicolas investigated algorithms that find the lower (or upper) median
> value.  The lower median is the median iff there are an odd number of
> entries in our list, or the lower of the central values in the sort,
> when there are an even number of values in the list. So, we need the
> upper _and_ lower median when there are an even number of entries.  I
> guess the necessity of doing those two related searches may change the
> relative strengths of the algorithms, but I'm sure this is a
> well-known problem.

My trivial solution to this was that since the data is now in two
"partitions", all one needs to do is quickly scan through the upper
partition to find the minimum value. Since you already have the lower
median from the select run, this minimum is by definition the upper
median. This can be averaged with the lower median and you are done.
Brain dead, perhaps, but it worked for my test. I did not (yet)
investigate whether this minimum can be located in some more expedient
fashion (i.e. did the select put the minimum in some consistent place
where we don't have to scan through half the input array?).

Mike

-- 
mike.ress...@alum.mit.edu
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PRNGs and multi-threading

[Robert Kern]

On Fri, Aug 21, 2009 at 20:50, Sturla Molden wrote:
> Sturla Molden skrev:
>> It seems there is a special version of the Mersenne Twister for this.
>> The code is LGPL (annoying for SciPy but ok for me).
>
> http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/DC/dgene.pdf
> 
>
> Basically it encodes the thread-ids in the characteristic polynomial of
> the MT, producing multiple small-period, independent MTs. That solves it
> then. Too bad this is LGPL. It would be a very useful enhancement to
> RandomState.

I agree. It might be possible to re-implement it from the original
papers, but it's a chunk of work.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Problems distributing NumPy

[Michael Cooper]

Hi all-

 

I am writing a C++ application with embedded Python scripting. Some of the
scripts use NumPy, so I have been working out the best way to distribute
NumPy with my software. At the moment, I've got a private folder which I add
to the Python path. In this folder, I include all the files which would
usually get installed to the "site-packages" folder. To get these files,
I've simply unzipped the distutils installer (for NumPy, I am using the "no
SSE" version at the moment), and copied the contents of the resulting
"PLATLIB" folder to my private folder.

 

For all the other libraries I am using, this method seems to work fine.
However, with NumPy, if I do things this way, when I call "import_array()",
it jumps back out of the calling function, skipping the rest of that
function, and continues from there. If I install NumPy in the usual way,
this does not happen. It seems like the NumPy initialization is failing when
I install into the private folder, but not if I use the normal installer.

 

Many of the users of my software aren't particularly Python savvy, so having
them install everything manually is not an option. I would like to avoid
having my own installer call external installers, sine that's confusing for
some users. Finally, if possible, I would like to avoid changing the user's
"Python26" folder, since I have no way of knowing what else might be relying
on its contents. Lots of searching on how to install third-party libraries
has led me to the "PLATLIB" method, so I'm at a bit of a loss as to what
else to try. Does anyone here know what might be going wrong?

 

I am using Python 2.6, Boost 1.38, and NumPy 1.3.0 on a Windows XP system.
The embedding program is written in C++, and compiled using Visual Studio
2005.

 

Thanks,

 

Michael

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerating NumPy computations [Was: GPU Numpy]

[Matthew Brett]

Hi,

> Indeed. In the future, if OpenCL is the way to go, it may even be
> helpful to have Numpy using OpenCL directly, as AMD provides an SDK
> for OpenCL, and with Larrabee approaching, Intel will surely provide
> one of its own.

I was just in a lecture by one of the Intel people about OpenCL:

http://parlab.eecs.berkeley.edu/bootcampagenda
http://parlab.eecs.berkeley.edu/sites/all/parlab/files/OpenCL_Mattson.pdf

He offered no schedule for an Intel OpenCL implementation, but said
that they were committed to it.

The lectures in general were effective in pointing out what a
time-consuming effort it can be moving algorithms into the the
parallel world - including GPUs.  The lecture just passed cited the
example of a CUDA-based BLAS implementation on the GPU that was slower
than the CPU version.Making BLAS go faster required a lot of work
to find optimal strategies for blocking, transfer between CPU / GPU
shared memory / GPU registers, vector sizes and so on - this on a
specific NVIDIA architecture.

I can imagine Numpy being useful for scripting in this
C-and-assembler-centric world, making it easier to write automated
testers, or even generate C code.

Is anyone out there working on this kind of stuff?  I ask only because
there seems to be considerable interest here on the Berkeley campus.

Best,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PRNGs and multi-threading

[Sturla Molden]

Sturla Molden skrev:
> It seems there is a special version of the Mersenne Twister for this. 
> The code is LGPL (annoying for SciPy but ok for me).

http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/DC/dgene.pdf 


Basically it encodes the thread-ids in the characteristic polynomial of 
the MT, producing multiple small-period, independent MTs. That solves it 
then. Too bad this is LGPL. It would be a very useful enhancement to 
RandomState.

Sturla Molden



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PRNGs and multi-threading

[Sturla Molden]

Robert Kern skrev:
> As long as you use different seeds, I believe this is fine. The state
> size of MT is so enormous that almost any reasonable use will not find
> overlaps.
>   

It seems there is a special version of the Mersenne Twister for this. 
The code is LGPL (annoying for SciPy but ok for me).

http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/DC/dc.html 



Sturla Molden
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] A better median function?

[Matthew Brett]

Hi,

> Nicolas Devillard discusses several algorithms at
> http://ndevilla.free.fr/median/median/index.html

Thanks for this.  A loud 'yes' from the back of the internet too.

I contacted Nicolas Devillard a year or so ago to ask him if we could
include his code in Scipy, and he said 'yes'.  I can forward this if
that's useful.

Nicolas investigated algorithms that find the lower (or upper) median
value.  The lower median is the median iff there are an odd number of
entries in our list, or the lower of the central values in the sort,
when there are an even number of values in the list. So, we need the
upper _and_ lower median when there are an even number of entries.  I
guess the necessity of doing those two related searches may change the
relative strengths of the algorithms, but I'm sure this is a
well-known problem.

Best,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] A better median function?

[Chad Netzer]

On Fri, Aug 21, 2009 at 8:47 AM, Mike Ressler wrote:
> I presented this during a lightning talk at the scipy conference
> yesterday, so again, at the risk of painting myself as a flaming
> idiot:
>
> -
> Wanted: A Better/Faster median() Function
>
> numpy implementation uses simple sorting algorithm:
> Sort all the data using the .sort() method
> Return middle value (or mean of two middle values)

Michael and I also discussed this briefly this morning at the SciPy
conference.  I'll summarize a bit:

scipy.signals has a medianfilter() implemented in C which uses the
Hoare selection algorithm:

http://en.wikipedia.org/wiki/Selection_algorithm#Partition-based_general_selection_algorithm

This *may* suit Michael's needs.  More generally, C++ std lib has both
partial_sort() and nth_element(), both of which would be "nice"
functionality to have natively (ie. C speeds) in numpy, imo:

http://www.cplusplus.com/reference/algorithm/partial_sort/
http://www.cplusplus.com/reference/algorithm/nth_element/

Since both can be made from a modified subset of quicksort, it should
be straightforward to craft these methods from the existing numpy
quicksort code.  With these primitives, it is trivial to improve the
existing numpy median() implementations.   I'll probably attempt to
tackle it myself, unless anyone else has done it (or has a better
idea).

Certainly, the ability to quickly find the minimal or maximal n
elements of a sequence, without having to perform a full sort, would
be of use to many numpy users.  Has this problem already been solved
in numpy?

-Chad
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Removing scipy.stsci was [Re: [SciPy-dev] Deprecate chararray [was Plea for help]]

[Stéfan van der Walt]

Hi Vincent

2009/8/21 Vincent Schut :
> I know it probably will be a pretty involved task, as ndimage comes from
> numarray and seems to be largely implemented in C. But I really wanted
> to raise the issue now the image processing subject turns up once again,
> and hope some folks with more/better programming skills than me might
> like the idea...

What would you like the behaviour to be?  For example, how should
ndimage.zoom handle these missing values?

Regards
Stéfan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PRNGs and multi-threading

[Sturla Molden]

Robert Kern skrev:
> Although you don't really have re-entrancy issues, you will usually
> want one PRNG per thread for determinism.
>   
I see... numpy.random.rand does not have re-entrancy issues because of 
the GIL, but I get indeterminism from the OS scheduling the threads. 
RandomState might not release the GIL either, but preserves determinism 
in presence of multiple threads. Thanks.  :-)
 

Regards,
Sturla Molden







___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] A better median function?

[David Goldsmith]

Ouch, didn't check my to address first, sorry!!!

DG

--- On Fri, 8/21/09, David Goldsmith  wrote:

> From: David Goldsmith 
> Subject: Re: [Numpy-discussion] A better median function?
> To: "Discussion of Numerical Python" 
> Date: Friday, August 21, 2009, 9:50 AM
> Not to make you regret your post ;-)
> but, you having readily furnished your email address, I'm
> taking the liberty of forwarding you my resume - I'm the guy
> who introduced himself yesterday by asking if you knew Don
> Hall - in case you have need of an experienced CCD data
> reduction programmer who knows Python, numpy, and
> matplotlib, as well as IDL, matlab, C/C++, and, from the
> "distant past" FORTRAN (not to mention advanced math and a
> little advanced physics, to boot).  Caveat: I'm not
> presently in a position to relocate. :-(  Thanks for
> your time and consideration,
> 
> David Goldsmith
> 
> DAVID GOLDSMITH
> 2036 Lakemoor Dr. SW
> Olympia, WA 98512
> 360-753-2318
> dgoldsmith...@alumni.brown.edu
> 
> Career Interests: Support of research possessing a strong
> component of one or more of the following: mathematics,
> statistics, programming, modeling, physical sciences,
> engineering, etc.
> 
> Desired salary rate: $75,000/yr.
> 
> 
> Skills
> 
>     Computer
> 
> Operating Systems: Windows, Macintosh, Unix 
> 
> Programming/Technical: Python, C/C++, SWIG, numpy,
> matplotlib, wxmpl, wxWidgets, SPE, Visual Studio .NET 2003,
> Trac, TortoiseSVN, RapidSVN, WinCVS, LAPACK, Matlab,
> Scientific Workplace, IDL, FORTRAN, Splus, Django (learning
> in progress).
> 
> Office: MS Word, Excel, PowerPoint, Outlook, Publisher,
> etc.; Page Maker; etc. 
> 
> Communications: Firefox, Thunderbird, VPN, MS Explorer,
> Netscape, NCSA Telnet, Fetch, WS FTP, telnet, ftp, lynx,
> pine, etc. 
> 
>     Other
> 
> Advanced mathematics, statistics, physics, fluid dynamics,
> engineering, etc.; technical documentation.
> 
> 
> Programming Employment
> 
> Technical Editor (Research Manager); June, 2009 to present;
> Planetary Sciences Group, Dept. of Physics, University of
> Central Florida, Orlando, FL (but working out of Olympia,
> WA).  Write and review a broad range of docstrings for
> NumPy, the standard Python module for numerical computing,
> and manage the 2009 NumPy Documentation Summer Marathon,
> including volunteer recruitment and coordination, project
> promotion, grant writing for perpetuation of the project,
> etc. 
> 
> Programming Mathematical Modeler (Functional Analyst II);
> June, 2004 through February, 2008; Emergency Response
> Division, National Oceanic and Atmospheric Administration,
> Seattle, WA (under contract with General Dynamics
> Information Technology, Fairfax, VA).  Develop 3D
> enhancements to existing 2D estuarine circulation codes and
> data visualization and analysis tools in Python and C++,
> using SWIG, numpy, C/LAPACK, ATLAS, matplotlib, wxmpl, SPE,
> Visual Studio/Visual C++, wxWidgets, RapidSVN, TortoiseSVN,
> WinCVS, etc. as development tools; confer regularly with
> other physical scientists, mathematicians, and programmers
> about these tools and other issues/projects related to
> hazardous material emergency response.
> 
> Programming Statistician (Research Associate V); May, 1999
> to September, 2001; Institute for Astronomy, University of
> Hawai`i, Hilo.  Developed IDL-based software for
> analysis of data obtained in development of solid-state
> sensor technology for the Next Generation Space Telescope,
> and other related computer activities.
> 
> Programming Research Assistant; September to December,
> 1997; Physics Dept., Univ. of Montana, Missoula. 
> Assisted in the development of a FORTRAN computational model
> for optimization of toroidal plasma confinement. 
> 
> Programming Research Assistant; June to August, 1997;
> Physics Dept., Univ. of Montana, Missoula.  Assisted in
> FORTRAN computer modeling of passive scalar transport in the
> stratosphere. 
> 
> Programming Research Assistant; June to August, 1997;
> Mathematical Sciences Dept., Univ. of Montana,
> Missoula.  Developed, in MATLAB, a
> cellular-automata-based simulation of flow around windmill
> turbine blades. 
> 
> Programming Consultant; April, 1995; Earth Justice Legal
> Defense Fund, Honolulu, Hawai`i.  Developed Excel
> spreadsheet to determine sewage discharge violations from
> municipal wastewater facility records.
> 
> Programming Research Assistant; June to August, 1985 and
> 1986; Plasma Physics Branch, Naval Research Laboratory,
> Washington, DC.  Assisted in FORTRAN computer modeling
> of plasma switching devices.
> 
> 
> Publications (abridged)
> 
> 2000, w/ D. Hall (1st author) et al., "Characterization of
> lambda_c ~ 5 micron Hg:Cd:Te Arrays for Low-Background
> Astronomy", Optical and IR Telescope Instrumentation and
> Detectors, Proceedings of SPIE, Vol. 4008, Part 2. 
> 
> 2000, w/ D. Hall (1st author) et al., "Molecular Beam
> Epitaxial Mercury Cadmium Telluride: A Quiet, Warm FPA For
> NGST", Astr. Soc. Pacific Co

Re: [Numpy-discussion] A better median function?

[David Goldsmith]

Not to make you regret your post ;-) but, you having readily furnished your 
email address, I'm taking the liberty of forwarding you my resume - I'm the guy 
who introduced himself yesterday by asking if you knew Don Hall - in case you 
have need of an experienced CCD data reduction programmer who knows Python, 
numpy, and matplotlib, as well as IDL, matlab, C/C++, and, from the "distant 
past" FORTRAN (not to mention advanced math and a little advanced physics, to 
boot).  Caveat: I'm not presently in a position to relocate. :-(  Thanks for 
your time and consideration,

David Goldsmith

DAVID GOLDSMITH
2036 Lakemoor Dr. SW
Olympia, WA 98512
360-753-2318
dgoldsmith...@alumni.brown.edu

Career Interests: Support of research possessing a strong component of one or 
more of the following: mathematics, statistics, programming, modeling, physical 
sciences, engineering, etc.

Desired salary rate: $75,000/yr.


Skills

Computer

Operating Systems: Windows, Macintosh, Unix 

Programming/Technical: Python, C/C++, SWIG, numpy, matplotlib, wxmpl, 
wxWidgets, SPE, Visual Studio .NET 2003, Trac, TortoiseSVN, RapidSVN, WinCVS, 
LAPACK, Matlab, Scientific Workplace, IDL, FORTRAN, Splus, Django (learning in 
progress).

Office: MS Word, Excel, PowerPoint, Outlook, Publisher, etc.; Page Maker; etc. 

Communications: Firefox, Thunderbird, VPN, MS Explorer, Netscape, NCSA Telnet, 
Fetch, WS FTP, telnet, ftp, lynx, pine, etc. 

Other

Advanced mathematics, statistics, physics, fluid dynamics, engineering, etc.; 
technical documentation.


Programming Employment

Technical Editor (Research Manager); June, 2009 to present; Planetary Sciences 
Group, Dept. of Physics, University of Central Florida, Orlando, FL (but 
working out of Olympia, WA).  Write and review a broad range of docstrings for 
NumPy, the standard Python module for numerical computing, and manage the 2009 
NumPy Documentation Summer Marathon, including volunteer recruitment and 
coordination, project promotion, grant writing for perpetuation of the project, 
etc. 

Programming Mathematical Modeler (Functional Analyst II); June, 2004 through 
February, 2008; Emergency Response Division, National Oceanic and Atmospheric 
Administration, Seattle, WA (under contract with General Dynamics Information 
Technology, Fairfax, VA).  Develop 3D enhancements to existing 2D estuarine 
circulation codes and data visualization and analysis tools in Python and C++, 
using SWIG, numpy, C/LAPACK, ATLAS, matplotlib, wxmpl, SPE, Visual 
Studio/Visual C++, wxWidgets, RapidSVN, TortoiseSVN, WinCVS, etc. as 
development tools; confer regularly with other physical scientists, 
mathematicians, and programmers about these tools and other issues/projects 
related to hazardous material emergency response.

Programming Statistician (Research Associate V); May, 1999 to September, 2001; 
Institute for Astronomy, University of Hawai`i, Hilo.  Developed IDL-based 
software for analysis of data obtained in development of solid-state sensor 
technology for the Next Generation Space Telescope, and other related computer 
activities.

Programming Research Assistant; September to December, 1997; Physics Dept., 
Univ. of Montana, Missoula.  Assisted in the development of a FORTRAN 
computational model for optimization of toroidal plasma confinement. 

Programming Research Assistant; June to August, 1997; Physics Dept., Univ. of 
Montana, Missoula.  Assisted in FORTRAN computer modeling of passive scalar 
transport in the stratosphere. 

Programming Research Assistant; June to August, 1997; Mathematical Sciences 
Dept., Univ. of Montana, Missoula.  Developed, in MATLAB, a 
cellular-automata-based simulation of flow around windmill turbine blades. 

Programming Consultant; April, 1995; Earth Justice Legal Defense Fund, 
Honolulu, Hawai`i.  Developed Excel spreadsheet to determine sewage discharge 
violations from municipal wastewater facility records.

Programming Research Assistant; June to August, 1985 and 1986; Plasma Physics 
Branch, Naval Research Laboratory, Washington, DC.  Assisted in FORTRAN 
computer modeling of plasma switching devices.


Publications (abridged)

2000, w/ D. Hall (1st author) et al., "Characterization of lambda_c ~ 5 micron 
Hg:Cd:Te Arrays for Low-Background Astronomy", Optical and IR Telescope 
Instrumentation and Detectors, Proceedings of SPIE, Vol. 4008, Part 2. 

2000, w/ D. Hall (1st author) et al., "Molecular Beam Epitaxial Mercury Cadmium 
Telluride: A Quiet, Warm FPA For NGST", Astr. Soc. Pacific Conf. Ser., Vol. 
207. 

1997, w/ A. Ware (1st author) et al., "Stability of Small Aspect Ratio Toroidal 
Hybrid Devices", American Physical Society, Plasma Physics Section, Semi-annual 
meeting. 


Education (abridged)

Master of Arts, Mathematical Sciences, University of Montana, Missoula, awarded 
May, 1998. GPA: 4.0. 

Master of Science, Aquacultural Engineering, University of Hawai`i, Manoa, 
awarded August, 1993. GPA: 3.72. 

Bachelor of Arts, Mathematics, B

Re: [Numpy-discussion] PRNGs and multi-threading

[Robert Kern]

On Fri, Aug 21, 2009 at 17:48, Sturla Molden wrote:
>
> I am not sure if this is the right place to discuss this issue. However,
> I problem I keep running into in Monte Carlo simulations is generating
> pseudorandom numbers with multiple threads. PRNGs such as the Mersenne
> Twister keep an internal state, which prevents the PRNG from being
> re-entrant and thread-safe.

C extension function calls are always atomic unless if they release
the GIL. numpy.random does not. You have de facto locks thanks to the
GIL.

> Possible solutions:
>
>
> 1. Use multiple instances of PRNG states (numpy.random.RandomState), one
> for each thread. This should give no contention, but is this
> mathematically acceptable? I don't know. At least I have not seen any
> proof that it is.

As long as you use different seeds, I believe this is fine. The state
size of MT is so enormous that almost any reasonable use will not find
overlaps.

Although you don't really have re-entrancy issues, you will usually
want one PRNG per thread for determinism.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] PRNGs and multi-threading

[Sturla Molden]

I am not sure if this is the right place to discuss this issue. However, 
I problem I keep running into in Monte Carlo simulations is generating 
pseudorandom numbers with multiple threads. PRNGs such as the Mersenne 
Twister keep an internal state, which prevents the PRNG from being 
re-entrant and thread-safe.


Possible solutions:


1. Use multiple instances of PRNG states (numpy.random.RandomState), one 
for each thread. This should give no contention, but is this 
mathematically acceptable? I don't know. At least I have not seen any 
proof that it is.



2. Protect the PRNG internally with a spinlock. In Windows lingo, that is:

#include 
static volatile long spinlock = 0;
#define ACQUIRE_SPINLOCK while(InterlockedExchangeAcquire(&spinlock, 1));
#define RELEASE_SPINLOCK InterlockedExchangeAcquire(&spinlock, 0);

Problem: possible contention between thread, idle work if threads spins 
a lot.



3. Use a conventional mutex object to protect the PRNG (e.g. 
threading.Lock in Python or CRITICAL_SECTION in Windows).  Problem: 
contention, context shifting, and mutexes tend to be slow. Possibly the 
worst solution.



4. Put the PRNG in a dedicated thread, fill up rather big arrays with 
pseudo-random numbers, and write them to a queue. Problem: Context 
shifting unless a CPU is dedicated to this task. Unless producing random 
numbers consitutes a major portion of the simulation, this should not 
lead to much contention.

import threading
import Queue
import numpy
from numpy.random import rand

class PRNG(threading.Thread):

''' a thread that generate arrays with random numbers
and dumps them to a queue '''

def __init__(self, nthreads, shape):
self.shape = shape
self.count = numpy.prod(shape)
self.queue = Queue.Queue(4 * nthreads) # magic number
self.stop_evt = threading.Event()
   
def generate(self, block=True, timeout=None):
return self.queue.get(block,timeout)
   
def join(self, timeout=None):
self.stop_evt.set()
super(PRNG,self).join(timeout)
   
def run(self):
tmp = rand(count).reshape(shape)
while 1:
try:
self.queue.put(tmp, block=True, timeout=2.0)
except Queue.Full:
if self.stop_evt.isSet(): break
else:
tmp = rand(count).reshape(shape)





Do you have any view on this? Is there any way of creating multiple 
independent random states that will work correctly? I know of SPRNG 
(Scalable PRNG), but it is made to work with MPI (which I don't use).


Regards,
Sturla Molden












___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] A better median function?

[Mike Ressler]

I presented this during a lightning talk at the scipy conference
yesterday, so again, at the risk of painting myself as a flaming
idiot:

-
Wanted: A Better/Faster median() Function

numpy implementation uses simple sorting algorithm:
Sort all the data using the .sort() method
Return middle value (or mean of two middle values)

One doesn’t have to sort all data – need only the middle value

Nicolas Devillard discusses several algorithms at
http://ndevilla.free.fr/median/median/index.html

Implemented Devillard’s version of the Numerical Recipes select()
function using ctypes: 2 to 20 times faster on the large (> 10^6
points) arrays I tested
--- Caveat: I don’t have all the bells and whistles of the built-in
median function (multiple dimensions, non-contiguous, etc.)

Any of the numpy developers interested in pursuing this further?
---

I got a fairly loud "yes" from the back of the room which a few of us
guessed was Robert Kern. I take that as generic interest at least in
checking this out.

The background on this is that I am doing some glitch finding
algorithms where I call median frequently. I think my ultimate problem
is not in median(), but how I loop through the data, but that is a
different discussion. What I noticed as I was investigating was what I
noted in the slide above. Returning the middle of a sorted vector is
not a bad thing to do (admit it, we've all done it at some point), but
it does too much work. Things that are lower or higher than the median
don't need to be in a perfectly sorted order if all we are after is
the median value.

I did some googling and came up with the web page noted above. I used
his modified NumRec select() function as an excuse to learn ctypes,
and my initial weak attempts were successful. The speed ups depend
highly on the length of the data and the randomness - things that are
correlated or partially sorted already go quickly. My caveat is that
my select-based median is too simple; it must have 1-d contiguous data
of a predefined type. It also moves the data in place, affecting the
original variable. I have no idea how this will blow up if implemented
in a general purpose way.

Anyway, I'm not enough of a C-coder to have any hope of improving this
to the point where it can be included in numpy itself. However, if
someone is willing to take up the torch, I will volunteer to assist
with discussion, prototyping a few routines, and testing (I have lots
of real-world data). One could argue that the current median
implementation is good enough (and it probably is for 99% of all
usage), but I view this as a chance to add an industrial strength
routine to the numpy base.

Thanks for listening.

Mike

-- 
mike.ress...@alum.mit.edu
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerating NumPy computations [Was: GPU Numpy]

[Matthieu Brucher]

>> I personally think that, in general, exposing GPU capabilities directly
>> to NumPy would provide little service for most NumPy users.  I rather
>> see letting this task to specialized libraries (like PyCUDA, or special
>> versions of ATLAS, for example) that can be used from NumPy.
>
> specialized library can be a good start as currently their is too much
> incertitude in the language(opencl vs nvidia api driver(pycuda, but not
> cublas, cufft,...) vs c-cuda(cublas, cufft))

Indeed. In the future, if OpenCL is the way to go, it may even be
helpful to have Numpy using OpenCL directly, as AMD provides an SDK
for OpenCL, and with Larrabee approaching, Intel will surely provide
one of its own.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerating NumPy computations [Was: GPU Numpy]

[Frédéric Bastien]

Hi,

On Thu, Aug 20, 2009 at 7:07 AM, Francesc Alted  wrote:

> El dj 20 de 08 del 2009 a les 00:37 -0700, en/na Erik Tollerud va
> escriure:
> > > NumPy arrays on the GPU memory is an easy task. But then I would have
> to
> > > write the computation in OpenCL's dialect of C99? But I'd rather
> program
> > > everything in Python if I could. Details like GPU and OpenCL should be
> > > hidden away. Nice looking Python with NumPy is much easier to read and
> > > write. That is why I'd like to see a code generator (i.e. JIT compiler)
> > > for NumPy.
> >
> > This is true to some extent, but also probably difficult to do given
> > the fact that paralellizable algorithms are generally more difficult
> > to formulate in striaghtforward ways.  In the intermediate-term, I
> > think there is value in having numpy implement some sort of interface
> > to OpenCL or cuda - I can easily see an explosion of different
> > bindings (it's already starting), and having a "canonical" way encoded
> > in numpy or scipy is probably the best way to mitigate the inevitable
> > compatibility problems... I'm partial to the way pycuda can do it
> > (basically, just export numpy arrays to the GPU and let you write the
> > code from there), but the main point is to just get some basic
> > compatibility in pretty quickly, as I think this GPGPU is here to
> > stay...
>
> Maybe.  However I think that we should not forget the fact that, as
> Stula pointed out, the main bottleneck for *many* problems nowadays is
> memory access, not CPU speed.  GPUs may have faster memory, but only a
> few % better than main stream memory.  I'd like to hear from anyone here
> having achieved any kind of speed-up in their calculations by using GPUs
> instead of CPUs.  By looking at these scenarios we may get an idea of
> where GPUs can be useful, and if driving an effort for give support for
> them in NumPy would be worth the effort.


I have around 10x speed up in convolution. I compare again my own version on
the cpu that is 20-30x faster then the version in scipy... I should backport
some of my optimisation(not all possible as I remove some case), but I
didn't get the time. The GPU are the most usefull when the bottleneck is the
cpu, not the memory and the probleme must be highly parallel. In that case
reported speed-up of 100-200x have been reported. But take those number with
a grain of salt, many of them don't talk much about the cpu implementation.
In that case, they probable compare a highly optimized version on the GPU
again a not optimised version on the CPU. I have see a case where they don't
tell withch version of blas they used on the cpu for matrix multiplication.
So this can be that they just forget to tell that they used an optimized one
or that they didn't used them. In the last case, the speed-up don't have any
meaning...


>
>
> I personally think that, in general, exposing GPU capabilities directly
> to NumPy would provide little service for most NumPy users.  I rather
> see letting this task to specialized libraries (like PyCUDA, or special
> versions of ATLAS, for example) that can be used from NumPy.


specialized library can be a good start as currently their is too much
incertitude in the language(opencl vs nvidia api driver(pycuda, but not
cublas, cufft,...) vs c-cuda(cublas, cufft))

One think that could help all those specialized libraries(I make one with
James B. cuda_ndarray) is to have a standardized version of NDarray for the
gpu. But I'm not shure it is a good time to do it now.


>
>
> Until then, I think that a more direct approach (and one that would
> deliver results earlier) for speeding-up NumPy is to be aware of the
> hierachical nature of the different memory levels in current CPU's and
> make NumPy to play nicely with it.  In that sense, I think that applying
> the blocking technique (see [1] for a brief explanation) for taking
> advantage of both spatial and temporal localities is the way to go.  For
> example, most part of the speed-up that Numexpr achieves comes from the
> fact that it uses blocking during the evaluation of complex expressions.
> This is so because the temporaries are kept small and can fit in current
> CPU caches.  Implementing similar algorithms in NumPy should not be that
> difficult, most specially now that it already exists the Numexpr
> implementation as a model.
>
> And another thing that may further help to fight memory slowness (or
> CPU/GPU quickness, as you prefer ;-) in the next future is compression.
> Compression already helped bringing data faster from disk to CPU in the
> last 10 years, and now, it is almost time that this can happen with the
> memory too, not only disk.
> In [1] I demonstrated that compression can *already* help transmitting
> data in memory to CPU.  Agreed, right now this is only true for highly
> compressible data (which is an important corner case anyway), but in the
> short future we will see how the compression technique would be able to
> accelerate computati

Re: [Numpy-discussion] itemsize() doesn't work

[Pauli Virtanen]

Fri, 21 Aug 2009 04:47:09 +, David Goldsmith wrote:
[clip]
> > > http://numpy.scipy.org/numpydoc/numpy-10.html
[clip]
> Is this editable through the Wiki?  I went to the
> Docstrings page and searched for "numpydoc" and "tutorial" and got no
> hits.

This is the old Numeric module documentation. It probably doesn't 
describe all points of Numpy accurately. Of course, the URL is 
misleading...

-- 
Pauli Virtanen

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Removing scipy.stsci was [Re: [SciPy-dev] Deprecate chararray [was Plea for help]]

[Vincent Schut]

Christopher Hanley wrote:
> Hi Stefan,
> 
> Never mind.  I just found the Sprint website and read the  
> description.  I'm sorry I hadn't found this sooner.  I would have made  
> plans to stay and help.  My apologizes.
> 
Hi list,

I just saw this too and would like to misuse this thread to suggest 
another enhancement related to image processing you might consider: make 
ndimage maskedarray aware. Unfortunately we don't have any money to 
spend, otherwise I'd love to support this financially too. But it's a 
thing I'm running into (ndimage not working with missing data, that is) 
regularly, and for which it often is pretty hard to work out a 
workaround. E.g. any of the resampling (ndimage.zoom) or kernel 
filtering routines choke on array's with NaN's, and don't recognize 
masked arrays. Alas virtually all of the image data I process (satellite 
imagery) contains missing/bad data...

I know it probably will be a pretty involved task, as ndimage comes from 
numarray and seems to be largely implemented in C. But I really wanted 
to raise the issue now the image processing subject turns up once again, 
and hope some folks with more/better programming skills than me might 
like the idea...

Oh and I know of course ndimage is scipy, and this list is numpy. But as 
the image processing subject emerged here, well...

Cheers,
Vincent Schut.

> Sorry,
> Chris
> 
> 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion