Re: [Numpy-discussion] NumPy 1.12.0 release

2017-01-17 Thread Jerome Kieffer 
On Tue, 17 Jan 2017 08:56:42 -0500
Neal Becker  wrote:

> I've installed via pip3 on linux x86_64, which gives me a wheel.  My 
> question is, am I loosing significant performance choosing this pre-built 
> binary vs. compiling myself?  For example, my processor might have some more 
> features than the base version used to build wheels.

Hi,

I have done some benchmarking (%timeit) for my code running in a
jupyter-notebook within a venv installed with pip+manylinux wheels
versus ipython and debian packages (on the same computer).
I noticed the debian installation was ~20% faster.

I did not investigate further if those 20% came from the manylinux (I
suspect) or from the notebook infrastructure.

HTH,
-- 
Jérôme Kieffer

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ufunc for sum of squared difference

2016-11-16 Thread Jerome Kieffer 
On Mon, 14 Nov 2016 22:38:25 +0200
eat  wrote:

> but it's not so obvious what's happening "under the hoods". Consider this
> (with an old Win7 machine):
> Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64
> bit (AMD64)]
> np.__version__
> '1.11.1'

What matters is the "blas" library used under the hood, hence the
options passed to numpy at compile time. I notices 20x differences
depending on the blas version.

But more importantly:
* results are the same (at the limit of the numerical precision)
* dot() was always faster than sum(square()), varying from a bit to a lot.

I agree this may change in future version of numpy.

Cheers,
-- 
Jérôme Kieffer
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ufunc for sum of squared difference

2016-11-14 Thread Jerome Kieffer 
On Fri, 11 Nov 2016 11:25:58 -0500
Matthew Harrigan  wrote:

> I started a ufunc to compute the sum of square differences here
> .
> It is about 4x faster and uses half the memory compared to
> np.sum(np.square(x-c)). 

Hi Matt,

Using *blas* you win already a factor two (maybe more depending on you blas 
implementation):

% python -m timeit -s "import numpy as np;x=np.linspace(0,1,int(1e7))" 
"np.sum(np.square(x-2.))"
10 loops, best of 3: 135 msec per loop

% python -m timeit -s "import numpy as np;x=np.linspace(0,1,int(1e7))" 
"y=x-2.;np.dot(y,y)"
10 loops, best of 3: 70.2 msec per loop


Cheers,
-- 
Jérôme Kieffer
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Build broken

2015-12-15 Thread Jerome Kieffer 

Hi,

I noticed the same kind of glitches for my project "pyFAI".
It seems some Travis VM are bugged.

Cheers,

Jerome
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-28 Thread Jerome Kieffer 
On Tue, 27 Oct 2015 15:35:50 -0700 (PDT)
"Juan Nunez-Iglesias"  wrote:

> Can someone here who understands more about distribution maybe write a blog 
> post detailing:

Hi,

Olivier Grisel from sklearn gave a very good talk on this topic at PyCon, 
earlier
this year:
http://www.pyvideo.org/video/3473/build-and-test-wheel-packages-on-linux-osx-win

Very instructive.
-- 
Jérôme Kieffer
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] [JOBs] Data analysis position at the European synchrotron

2015-06-13 Thread Jerome Kieffer 
Dear Pythonistas,

The European Synchrotron, ESRF, located in the French Alps, just got
approved a large upgrade in which data-analysis is a key element. I am
pleased to announce this strategy is built around Python and all code
developed in this frame will be based around Python and made
open-source. Feel free to distribute this around.

1 metadata manager position:
http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=413

2 data scientist positions:
http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=421
http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=414

3 software engineer positions
http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=418
http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=417
http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=419

Other related data analysis positions
http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=420
http://esrf.profilsearch.com/recrute/fo_annonce_voir.php?id=411

Best regards,

Jerome Kieffer
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] read not byte aligned records

2015-05-04 Thread Jerome Kieffer 
Hi, 
If you want to play with 10 bits data-blocks, read 5 bytes and work with 4 
entries at a time...

-- 
Jérôme Kieffer
Data analysis unit - ESRF
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: pyMIC v0.5 released

2015-03-27 Thread Jerome Kieffer 

Hi,

Interesting project. How close is the C++ kernel needed from OpenCL kernels ?
Is it directly portable ?

I have tested my OpenCL code (via pyopencl) on the Phi and I did not
get better performances than the dual-hexacore Xeon (i.e. ~2x slower than a 
GPU). 

Cheers
-- 
Jérôme Kieffer
Data analysis unit - ESRF
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Installation on Windows

2015-03-20 Thread Jerome Kieffer 
On Fri, 20 Mar 2015 09:45:41 +0100
Per Tunedal  wrote:

> Hi,
> how do I install Numpy on Windows? I've tried the setup.py file, but get
> an error message:
> 
> setup.py install
> 
> gives:
> No module named msvccompiler in numpy.distutils; trying from distutils
> error: Unable to find vcvarsall.bat
> 

get the ms compiler from
http://aka.ms/vcpython27
-- 
Jérôme Kieffer
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Rewrite np.histogram in c?

2015-03-16 Thread Jerome Kieffer 
On Mon, 16 Mar 2015 06:56:58 -0700
Jaime Fernández del Río  wrote:

> Dispatching to a different method seems like a no brainer indeed. The
> question is whether we really need to do this in C.

I need to do both unweighted & weighted histograms and we got a factor 5 using 
(simple) cython:
it is in the proceedings of Euroscipy, last year.
http://arxiv.org/pdf/1412.6367.pdf

We got much faster but that's another story.

In fact, many people coming from IDL or Matlab are surprised by the
poor performances of numpy's histogram.

Cheers

-- 
Jérôme Kieffer
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Bilinear interpolation (numpy only)

2014-12-14 Thread Jerome Kieffer 
On Sun, 14 Dec 2014 12:52:03 +0100
"Nicolas P. Rougier"  wrote:

> 
> Thanks Jérôme, I will look into your code. Having other filter might be 
> useful for my case.
> 
> While looking for code, I've also found this (pure python) implementation:
> http://stackoverflow.com/questions/12729228/simple-efficient-bilinear-interpolation-of-images-in-numpy-and-python

Great, I keep the link as well. it is interesting to have it.

The only drawback is the poor cache efficiency of the numpy implementation 
(where actually cython rocks)

Cheers,

-- 
Jérôme Kieffer
Data analysis unit - ESRF
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Bilinear interpolation (numpy only)

2014-12-14 Thread Jerome Kieffer 
On Sat, 13 Dec 2014 16:53:06 +0100
"Nicolas P. Rougier"  wrote:

> 
> Hi all,
> 
> Does anyone has a simple 2D linear interpolation for resizing an image 
> (without scipy) ?
> 
> Ideally, something like ```def zoom(Z, ratio): ...``` where Z is a 2D scalar 
> array and ratio the scaling factor.
> (I'm currently using ```scipy.ndimage.interpolation.zoom``` but I would like 
> to avoid the scipy dependency)

Hi Nicolas,

I have a self-contained cython class for that:
https://github.com/kif/pyFAI/blob/master/src/bilinear.pyx

The formula for bilinear interpolation in implemented there but it
needs some additionnal work for what you want to do.

Beside this I implemented an antialiased downscaler using Lanczos (order 1, 2 
or 3)
https://github.com/kif/imagizer/blob/qt/src/down_sampler.pyx.

Create a downscaler:
ds = down_sampler.DownScaler()
scaled_img = ds.scale(img, 4.5)

In this case the interpolation will be done on a vinicy of (2*4.5*3+1) pixel in 
the input image (and 2*3+1 in the output image) as it is doing Lanczos 3 by 
default.
This implementation is 2x faster than the Antialiased downscaler in PIL.

Cheers,

-- 
Jérôme Kieffer
Data analysis unit - ESRF
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] FFTS for numpy's FFTs (was: Re: Choosing between NumPy and SciPy functions)

2014-10-28 Thread Jerome Kieffer 
On Tue, 28 Oct 2014 04:28:37 +
Nathaniel Smith  wrote:

> It's definitely attractive. Some potential issues that might need dealing
> with, based on a quick skim:

In my tests, numpy's FFTPACK isn't that bad considering 
* (virtually) no extra overhead for installation
* (virtually) no plan creation time
* not that slower for each transformation

Because the plan creation was taking ages with FFTw, numpy's FFTPACK was often 
faster (overall)

Cheers,
-- 
Jérôme Kieffer
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ENH IncrementalWriter for .npy files

2014-09-02 Thread Jerome Kieffer 

Hi,

This feature is very similar to what is available in hdf5 and exposed
under h5py using chunks and max_size ...


Cheers,

-- 
Jérôme Kieffer
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy en français

2014-06-23 Thread Jerome Kieffer 
Hi,

As the request is is French, I will answer in French...



On Sun, 22 Jun 2014 22:01:43 +0200
pi8L4vion  wrote:

> Bonsoir à tous,
> 
> Est-ce qu’il y a des francophones qui pourraient échanger avec moi ?

Je suis Français ... par contre je ne connais pas de liste de
discussion sur numpy ni sur python scientifique en Français.

Je me pose d'ailleurs la question de la pertinence d'une telle liste
puisque toute la documentation est en anglais et que la communauté est
déjà assez limité, ce n'est pas en la fragmentant que cela va aider.

Mon conseil serait donc de communiquer en anglais (quitte à s'excuser
de la mauvaise qualité de l'expression anglaise: la plupart des gens
ici ne sont pas anglophones et comprendrons)

Bien cordialement
-- 
Jérôme Kieffer
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is there a pure numpy recipe for this?

2014-03-27 Thread Jerome Kieffer 
On Thu, 27 Mar 2014 16:19:54 +
"Aaron O'Leary"  wrote:

> 
> You might want to look at hdf5 if you're routinely running out of ram.
> I'm using h5py with multi gigabyte data on an ssd right now. It is very
> fast. You still have to be careful with your computations and try to
> avoid creating copies though.

Both for h5py and for memmapped files ... switching from windows to
linux are likely to help ...

-- 
Jérôme Kieffer
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] MKL + CPU, GPU + cuBLAS comparison

2013-11-26 Thread Jerome Kieffer 
On Tue, 26 Nov 2013 01:02:40 -0800
"Dinesh Vadhia"  wrote:

> Probably a loaded question but is there a significant performance difference 
> between using MKL (or OpenBLAS) on multi-core cpu's and cuBLAS on gpu's.  
> Does anyone have recent experience or link to an independent benchmark?
> 

Using Numpy (Xeon 5520 2.2GHz):

In [1]: import numpy
In [2]: shape = (450,450,450)
In [3]: start=numpy.random.random(shape).astype("complex128")
In [4]: %timeit result = numpy.fft.fftn(start)
1 loops, best of 3: 10.2 s per loop

Using FFTw (8 threads (2x quad cores):

In [5]: import fftw3
In [7]: result = numpy.empty_like(start)
In [8]: fft = fftw3.Plan(start, result, direction='forward', flags=['measure'], 
nthreads=8)
In [9]: %timeit fft()
1 loops, best of 3: 887 ms per loop

Using CuFFT (GeForce Titan):
1) with 2 transfers:
In [10]: import pycuda,pycuda.gpuarray as gpuarray,scikits.cuda.fft as 
cu_fft,pycuda.autoinit
In [11]: cuplan = cu_fft.Plan(start.shape, numpy.complex128, numpy.complex128)
In [12]: d_result = gpuarray.empty(start.shape, start.dtype)
In [13]: d_start = gpuarray.empty(start.shape, start.dtype)
In [14]: def cuda_fft(start):
   : d_start.set(start)
   : cu_fft.fft(d_start, d_result, cuplan)
   : return d_result.get()
   : 
In [15]: %timeit cuda_fft(start)
1 loops, best of 3: 1.7 s per loop

2) with 1 transfert:
In [18]: def cuda_fft_2():
cu_fft.fft(d_start, d_result, cuplan)
return d_result.get()
   : 
In [20]: %timeit cuda_fft_2()
1 loops, best of 3: 1.05 s per loop

3) Without transfer:
In [22]: def cuda_fft_3():
cu_fft.fft(d_start, d_result, cuplan)
pycuda.autoinit.context.synchronize()
   : 

In [23]: %timeit cuda_fft_3()
1 loops, best of 3: 202 ms per loop

Conclusion: 
A Geforce Titan (1000€) can be 4x faster than a couple of Xeon 5520 (2x 250€) 
if your data are already on the GPU.
Nota: Plan calculation are much faster on GPU then on CPU.
-- 
Jérôme Kieffer
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation

2013-09-18 Thread Jerome Kieffer 
On Wed, 18 Sep 2013 18:40:44 -0700
Antony Lee  wrote:

> Thanks, I didn't know that multiprocessing Managers could be used with
> processes not started by multiprocessing itself...  I will give them a try.
> I just need to compute FFTs, but speed is a real issue for me (I am using
> the results for real-time feedback).  To be honest I don't know yet if the
> FFTs are going to be the limiting step but I thought I may as well give
> pyFFTW a try and ran into that issue...

If FFTs are needed as well as speed, I got good results with PyCuda + CuFFT 
(and a GPU from Nvidia).

-- 
Jérôme Kieffer
On-Line Data analysis / Software Group 
ISDD / ESRF
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Error occurance on Skimage 0.9.0 version

2013-09-05 Thread Jerome Kieffer 
On Thu, 05 Sep 2013 17:35:16 +0400
Happyman  wrote:

>  Hello,
> 
> I downloaded skimage 0.9.0dev version and installed like following on Windows;
> cd C:\skimage\python setup.py install.
> 
> One error occurred: Unable to find vcvarsall.bat . It is not the first time I 
> have been taken such an error.
> What is the best way to avoid this problem???
> 
> Any answers will be appreciated

it looks like it is looking for the visual studio compiler ... do you have it ?
if not try mingw32 (only for windows32 bits) it often works.

Cheers,

-- 
Jérôme Kieffer
On-Line Data analysis / Software Group 
ISDD / ESRF
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] add .H attribute?

2013-07-23 Thread Jerome Kieffer 
On Tue, 23 Jul 2013 01:07:27 -0700
Fernando Perez  wrote:

> Silly suggestion: why not just make .H a callable?
> 
> a.H()

+1

-- 
Jérôme Kieffer
On-Line Data analysis / Software Group 
ISDD / ESRF
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] About speed vs math...

2013-06-03 Thread Jerome Kieffer 
On Mon, 3 Jun 2013 07:33:23 +0100
David Cournapeau  wrote:

> > While not surprising, I did not expect numpy to be so much slower (25x)...
> It is a known limitation of numpy scalars. As soon as you use array
> that are not tiny, the speed difference disappears and then favors
> numpy arrays (around 50 on my own machine, but that's platform
> specific).

Thanks for the figure and the explanation.


-- 
Jérôme Kieffer
On-Line Data analysis / Software Group 
ISDD / ESRF
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] About speed vs math...

2013-06-02 Thread Jerome Kieffer 
Hello,

I am giving some introduction tutorials to numpy and we notices a big
difference in speed between nuumpy and math for trigonometric
operations:

In [3]: %timeit numpy.sin(1)
10 loops, best of 3: 2.27 us per loop

In [4]: %timeit math.sin(1)
1000 loops, best of 3: 92.3 ns per loop

While not surprising, I did not expect numpy to be so much slower (25x)...

Is this normal ?
Cheers,
-- 
Jérôme Kieffer
Data analysis unit - ESRF
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Calling scipy blas f rom cython is extr emely slow

2013-02-23 Thread Jerome Kieffer 
On Sat, 23 Feb 2013 18:31:42 + (UTC)
Sergio Callegari  wrote:

> However, it is still slower than the cblas
> 
> cblas -> 0.69 sec
> scipy blas -> 0.74 sec

if you are using scipy blas, the real question is which blas is underneath ?
OpenBlas, GotoBlas, Atlas, MKL ?

Under Debian I observed a x17 in speed from 35s to 2s with an "apt-get install 
atlas" on Armando's code.

Cheers,

-- 
Jérôme Kieffer
Data analysis unit - ESRF
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [enhancement] sum_angle() and sum_polar()

2012-05-29 Thread Jerome Kieffer 
On Tue, 29 May 2012 10:03:04 -0700
Stéfan van der Walt  wrote:

> On Mon, May 28, 2012 at 11:53 AM, Travis Oliphant  wrote:
> > I could see these functions going into scipy.ndimage but again because they
> > are not necessarily just image processing functions, and the fact that they
> > are so simple, perhaps they are best put into NumPy itself.
> 
> I'm wondering about the general applicability of these functions.  Can
> anyone suggest some use cases?

I wrote a whole library about that ... pyFAI (available in debian)
https://forge.epn-campus.eu/attachments/1459/20111010-PyFAI-Poster-A0.pdf

Unfortunately real detector are never completely orthogonal to the incident 
beam, pixels are never square, ... what makes things more complicated.

Cheers,

-- 
Jérôme Kieffer
Data analysis unit - ESRF
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] fast method to to count a particular value in a large matrix

2012-02-04 Thread Jerome Kieffer 
On Sat, 4 Feb 2012 14:35:08 -0600
Benjamin Root  wrote:

> 
> no.unique() can return indices and reverse indices.  It would be trivial to
> histogram the reverse indices using np.histogram().

Even np.histogram(abc,unique_elem) or something like this. 
Works if unique_elem is ordered.

np.histogram(abc,list(unique_elem)+[unique_elem[-1]+1])[0].reshape(-1,1)
is 40x faster and gives the same result.

-- 
Jérôme Kieffer
Data analysis unit - ESRF
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion