Re: [Numpy-discussion] How a transition to C++ could work

Den 19.02.2012 11:30, skrev Christopher Jordan-Squire: > > Can this possibly be extended to the following: How will Mark's > (extensive) experience about performance and long-term consequences of > design decisions be communicated to future developers? We not only > want new numpy developers, we wa

Re: [Numpy-discussion] How a transition to C++ could work

Den 19. feb. 2012 kl. 09:51 skrev Stéfan van der Walt : > > OK, so let's talk specifics: how do you dynamically grab a function pointer > to a compiled C++ library, a la ctypes? Feel free to point me to > StackOverflow or elsewhere. > You declare the function with the signature extern "C".

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 19.02.2012 01:12, skrev Nathaniel Smith: > > I don't oppose it, but I admit I'm not really clear on what the > supposed advantages would be. Everyone seems to agree that >-- Only a carefully-chosen subset of C++ features should be used >-- But this subset would be pretty useful > I wond

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 18.02.2012 23:54, skrev Travis Oliphant: > Another factor. the decision to make an extra layer of indirection makes > small arrays that much slower. I agree with Mark that in a core library we > need to go the other way with small arrays being completely allocated in the > data-structure

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 19.02.2012 00:33, skrev Sturla Molden: > Or just write everything in Cython, even the core? That is, use memory view syntax and fused types for generics, and hope it is stable before we are done ;-) Sturla ___ NumPy-Discussion mailing list Nu

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 19.02.2012 00:09, skrev David Cournapeau: > There are better languages than C++ that has most of the technical > benefits stated in this discussion (rust and D being the most > "obvious" ones), What about Java? (compile with GJC for CPython) Or just write everything in Cython, even the core

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 19.02.2012 00:09, skrev David Cournapeau: > reasons: knowledge, availability on "esoteric" platforms, etc… A new > language is completely ridiculous. Yes, that is why I argued against Cython as well. Personally I prefer C++ to C, but only if it is written in a readable way. And if the purpos

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 18.02.2012 23:24, skrev David Cournapeau: > Iterators as we have it in NumPy is something that is clearly limited > by C. Computers tend to have more than one CPU now. Iterators are inherently bad, whether they are written in C or C++. NumPy core should be written with objects that are scal

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 18.02.2012 22:25, skrev Benjamin Root: > 2.) My personal preference is an incremental refactor over to C++ > using STL, however, I have to be realistic. First, the exception > issue is problematic (unsolvable? I don't know). Second, one of > Numpy/Scipy's greatest strengths is the relative

Re: [Numpy-discussion] The end of numpy as we know it ?

Den 18. feb. 2012 kl. 17:12 skrev Alan G Isaac : > > > How does "stream-lined" code written for maintainability > (i.e., with helpful comments and tests) become *less* > accessible to amateurs?? I think you missed the irony. Sturla ___ NumPy-Disc

Re: [Numpy-discussion] Proposed Roadmap Overview

> > Albeit Cython has a special syntax for NumPy arrays, we are talking about > implementation of NumPy, not using it. I would not consider Cython for this > before e.g. memoryviews have been stable for a long period. The subset of > Cython we could safely use is not better than plain C. > >

Re: [Numpy-discussion] Proposed Roadmap Overview

> > I just meant what Sturla said, nothing more: > > "Cython is still 0.16, it is still unfinished. We cannot base NumPy on > an unfinished compiler." > Albeit Cython has a special syntax for NumPy arrays, we are talking about implementation of NumPy, not using it. I would not consider Cython

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 18. feb. 2012 kl. 14:38 skrev David Cournapeau : > I took a superficial look at zeromq 2.x sources: it looks like they don't use > much of the stl (beyond vector and some trivial usages of algorithm). I > wonder if this is linked ? > > FWIW, I would be fine with using such a subset in num

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 18. feb. 2012 kl. 05:56 skrev Charles R Harris : > > > But won't a C++ wrapper catch that? A try-catch block with MSVC will register an SEH with the operating system. GCC (g++) implements exceptions without SEH. What happens if GCC code tries to catch a std::bad_alloc? Windows intervene

Re: [Numpy-discussion] Proposed Roadmap Overview

> > Why would you even see an exception if it is caught before it escapes? I > would expect the C API to behave just as it currently does. What am I > missing? Structured exception handling in the OS. MSVC uses SEH for C++ exceptions. Memory allocation fails in gcc code. Instead of retur

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 18. feb. 2012 kl. 05:23 skrev Jason Grout : > On 2/17/12 10:10 PM, Sturla Molden wrote: > >> Sure. They just keep adding features for the expence of stability. No >> focus or sence of direction. Focus on a small feature set, make it >> right, then don't add t

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 18. feb. 2012 kl. 05:01 skrev Jason Grout : > On 2/17/12 9:54 PM, Sturla Molden wrote: >> We would have to write a C++ programming tutorial that is based on Pyton >> knowledge instead of C knowledge. > > I personally would love such a thing. It's been a whi

Re: [Numpy-discussion] Proposed Roadmap Overview

> > For what it's worth, Cython supports C++ now. I'm sure there are people > on this list that know much better than me the extent of this support, > so I will let them chime in, but here are some docs on it: > > http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html > Sure. They jus

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 18. feb. 2012 kl. 04:27 skrev Jason Grout : > On 2/17/12 9:07 PM, Sturla Molden wrote: >> >> Den 18. feb. 2012 kl. 01:58 skrev Charles R Harris >> mailto:charlesr.har...@gmail.com>>: >> >>> >>> >>> On Fri, Feb 17, 2012 at 4:4

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 17. feb. 2012 kl. 18:52 skrev Eric Firing :. > > It's true that matplotlib relies heavily on C++, both via the Agg > library and in its own extension code. Personally, I don't like this; I > think it raises the barrier to contributing. C++ is an order of > magnitude more complicated than

Re: [Numpy-discussion] Proposed Roadmap Overview

> > How about a different variation. How many people writing Python would happily > give up the following: > > 1) lists > 2) dictionaries > 3) default types > 4) classes > 5) automatic dellocation of memory > 1) std::vector 2) std::unordered_map 3) auto 4) class 5) std::shared_ptr Sturla

Re: [Numpy-discussion] Proposed Roadmap Overview

> . > > To be used approprietly, c++ requires much more discipline than c. Doing this > for a community-based project is very hard. Doing this with people who often > are scientist first and programmers second even harder. > This is very important. I am not sure it is doable. Bad C++ is far wo

Re: [Numpy-discussion] Proposed Roadmap Overview

Den 18. feb. 2012 kl. 01:58 skrev Charles R Harris : > > > On Fri, Feb 17, 2012 at 4:44 PM, David Cournapeau wrote: > I don't think c++ has any significant advantage over c for high performance > libraries. I am not convinced by the number of people argument either: it is > not my experience

Re: [Numpy-discussion] numpy.arange() error?

Den 9. feb. 2012 kl. 22:44 skrev eat : > > Maybe this issue is raised also earlier, but wouldn't it be more consistent > to let arange operate only with integers (like Python's range) and let > linspace handle the floats as well? > > Perhaps. Another possibility would be to let arange take

Re: [Numpy-discussion] Moving to gcc 4.* for win32 installers ?

On 07.02.2012 18:38, Sturla Molden wrote: > One potential problem I just discovered is dependency on a DLL called > libpthreadGC2.dll. This is not correct!!! :-D Two threading APIs can be used for OpenBLAS/GotoBLAS2, Win32 threads or OpenMP. driver/others/blas_server_omp.c

Re: [Numpy-discussion] Logical indexing and higher-dimensional arrays.

On 08.02.2012 18:17, josef.p...@gmail.com wrote: > I think I use it quite a bit, and I like that the broadcasting in > indexing is as flexible as the broadcasting of numpy arrays > themselves. > > x[np.arange(len(x)), np.arange(len(x))] gives the diagonal for example. Personally I would prefer

Re: [Numpy-discussion] Logical indexing and higher-dimensional arrays.

On 08.02.2012 15:49, Travis Oliphant wrote: > This sort of thing would take time, but is not out of the question in my mind > because I suspect the number of users and use-cases of "broadcasted" > fancy-indexing is small. In Matlab this (misfeature?) is generally used to compensate for the lac

Re: [Numpy-discussion] Logical indexing and higher-dimensional arrays.

On 08.02.2012 15:11, Olivier Delalleau wrote: > From a user perspective, I would definitely prefer cross-product > semantics for fancy indexing. In fact, I had never used fancy indexing > with more than one array index, so the behavior described in this thread > totally baffled me. If for instanc

Re: [Numpy-discussion] Logical indexing and higher-dimensional arrays.

On 08.02.2012 06:01, Travis Oliphant wrote: > Recall that the shape of the output with fancy indexing is determined by > broadcasting together the indexing objects and using that as the shape of the > output: > > x[ind1, ind2] will produce an output with the shape of "broadcast(ind1, > ind2)" w

Re: [Numpy-discussion] Logical indexing and higher-dimensional arrays.

On 07.02.2012 19:24, Sturla Molden wrote: > On 07.02.2012 19:17, Benjamin Root wrote: > >> >>> print x.shape >> (2, 3, 4) >> >>> print x[0, :, :].shape >> (3, 4) >> >>> print x[0, :, idx].shape >> (2, 3) > > Tha

Re: [Numpy-discussion] Logical indexing and higher-dimensional arrays.

On 07.02.2012 19:17, Benjamin Root wrote: > >>> print x.shape > (2, 3, 4) > >>> print x[0, :, :].shape > (3, 4) > >>> print x[0, :, idx].shape > (2, 3) That looks like a bug to me. The length of the first dimension should be the same. Sturla __

Re: [Numpy-discussion] Moving to gcc 4.* for win32 installers ?

On 07.02.2012 17:15, David Cournapeau wrote: > I did not know GotoBLAS2 was open source (it wasn't last time I > checked). That's very useful information, I will look into it. One potential problem I just discovered is dependency on a DLL called libpthreadGC2.dll. First, it's a DLL that must be

Re: [Numpy-discussion] avoiding loops when downsampling arrays

> for i in range(m): > for j in range(n): > found[i//4,j//4] = cond(x[i,j]) > Blah, that should be found[i//4,j//4] |= cond(x[i,j]) Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org h

Re: [Numpy-discussion] Moving to gcc 4.* for win32 installers ?

On 07.02.2012 17:14, David Cournapeau wrote: > How did you link a library with mixed C and gfortran ? Use gfortran instead of gcc when you link. gfortran knows what to do (and don't put -lgfortran in there). Something like this I think: gfortran -o whatever.pyd -shared cobj.o fobj.o -lmsvcr90 -

Re: [Numpy-discussion] avoiding loops when downsampling arrays

On 07.02.2012 15:27, eat wrote: > This is elegant and very fast as well! Just be aware that it depends on C ordered input. So: m,n = data.shape cond = lamda x : (x >= t1) & (x <= t2) x = cond(np.ascontiguousarray(data)).reshape((m//4, 4, n//4, 4)) found = np.any(np.any(x, axis=1)

Re: [Numpy-discussion] Moving to gcc 4.* for win32 installers ?

On 07.02.2012 14:38, Sturla Molden wrote: > May I suggest GotoBLAS2 instead of ATLAS? Or OpenBLAS, which is GotoBLAS2 except it is still maintained. https://github.com/xianyi/OpenBLAS ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org h

Re: [Numpy-discussion] Moving to gcc 4.* for win32 installers ?

On 27.10.2011 15:02, David Cournapeau wrote: >- we need to recompile atlas (but I can take care of it) May I suggest GotoBLAS2 instead of ATLAS? Is is faster (comparable to MKL), easier to build, and now released under BSD licence. http://www.tacc.utexas.edu/tacc-projects/gotoblas2 Sturl

Re: [Numpy-discussion] Moving to gcc 4.* for win32 installers ?

On 27.10.2011 15:02, David Cournapeau wrote: >- we need to recompile atlas (but I can take care of it) >- the biggest: it is difficult to combine gfortran with visual > studio (more exactly you cannot link gfortran runtime to a visual > studio executable). Why is that? I have used gfortr

Re: [Numpy-discussion] Moving to gcc 4.* for win32 installers ?

On 04.02.2012 16:55, Ralf Gommers wrote: > Although not ideal, I don't have a problem with that in principle. > However, wouldn't it break installing without admin rights if Python > is installed by the admin? Not on Windows. Sturla ___ Num

Re: [Numpy-discussion] avoiding loops when downsampling arrays

On 06.02.2012 22:27, Sturla Molden wrote: > > >> >> # Make a 4D view of this data, such that b[i,j] >> # is a 2D block with shape (4,4) (e.g. b[0,0] is >> # the same as a[:4, :4]). >> b = as_strided(a, shape=(a.shape[0]/4, a.shape[1]/4, 4, 4), >>

Re: [Numpy-discussion] avoiding loops when downsampling arrays

> > # Make a 4D view of this data, such that b[i,j] > # is a 2D block with shape (4,4) (e.g. b[0,0] is > # the same as a[:4, :4]). > b = as_strided(a, shape=(a.shape[0]/4, a.shape[1]/4, 4, 4), >strides=(4*a.strides[0], 4*a.strides[1], a.strides[0], > a.strides[1])) > Yes :-) B

Re: [Numpy-discussion] avoiding loops when downsampling arrays

The last t1 on each lineis of course t2. Sorry for the typo. Hard to code on an ipad ;-) Sturla Sendt fra min iPad Den 6. feb. 2012 kl. 22:12 skrev Sturla Molden : > > Something like this: > > m,n = data.shape > x = data.reshape((m,n//4,4)) > z = (x[0::4,...] >= t1)

Re: [Numpy-discussion] avoiding loops when downsampling arrays

found = np.any(z, axis=2) Sturla Sendt fra min iPad Den 6. feb. 2012 kl. 21:57 skrev Sturla Molden : > Short answer: Create 16 view arrays, each with a stride of 4 in both > dimensions. Test them against the conditions and combine the tests with an |= > operator. Thus you

Re: [Numpy-discussion] avoiding loops when downsampling arrays

Short answer: Create 16 view arrays, each with a stride of 4 in both dimensions. Test them against the conditions and combine the tests with an |= operator. Thus you replace the nested loop with one that has only 16 iterations. Or reshape to 3 dimensions, the last with length 4, and you can do

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

Den 24.01.2012 17:19, skrev David Warde-Farley: > > Hmm. Seeing as the width of a C long is inconsistent, does this imply that > the random number generator will produce different results on different > platforms? If it does, it is a C programming mistake. C code should never depend on the exact

Re: [Numpy-discussion] Cross-covariance function

Den 26.01.2012 17:25, skrev Pierre Haessig: > However, in the case this change is not possible, I would see this > solution : > * add and xcov function that does what Elliot and Sturla and I > described, because The current np.cov implementation returns the cross-covariance the way it is commonly

[Numpy-discussion] OT: MS C++ AMP library

The annoying part is, with this crap there will never be a standard OpenCL DLL in Windows. Sturla Molden ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 23:30, David Warde-Farley wrote: > I've figured it out. In numpy/core/src/multiarray/mapping.c, PyArray_GetMap > is using an int for a counter variable where it should be using an npy_intp. > > I've filed a pull request at https://github.com/numpy/numpy/pull/188 with a > regression t

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 10:15, Robert Kern wrote: > There are two different uses of long that you need to distinguish. One > is for sizes, and one is for parameters and values. The sizes should > definitely be upgraded to npy_intp. The latter shouldn't; these should > remain as the default integer type of P

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 10:16, Robert Kern wrote: > I'm sorry, what are you demonstrating there? Both npy_intp and C long are used for sizes and indexing. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 06:32, Sturla Molden wrote: > Den 24.01.2012 06:00, skrev Sturla Molden: >> Both i and length could overflow here. It should overflow on >> allocation of more than 2 GB. There is also a lot of C longs in the >> internal state (line 55-105), as well as the other f

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 09:21, Sturla Molden wrote: > randomkit.c handles C long correctly, I think. There are different codes > for 32 and 64 bit C long, and buffer sizes are size_t. distributions.c take C longs as parameters e.g. for the binomial distribution. mtrand.pyx correctly handles this,

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 06:32, Sturla Molden wrote: > The use of C long affects all the C and Pyrex source code in mtrand > module, not just mtrand.pyx. All of it is fubar on Win64. randomkit.c handles C long correctly, I think. There are different codes for 32 and 64 bit C long, and buffer siz

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

Den 24.01.2012 06:00, skrev Sturla Molden: > Both i and length could overflow here. It should overflow on > allocation of more than 2 GB. There is also a lot of C longs in the > internal state (line 55-105), as well as the other functions. The use of C long affects all the C and Pyr

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

Den 23.01.2012 22:08, skrev Christoph Gohlke: > > Maybe this explains the win-amd64 behavior: There are a couple of places > in mtrand where array indices and sizes are C long instead of npy_intp, > for example in the randint function: > >

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

Den 23.01.2012 22:08, skrev Christoph Gohlke: > Maybe this explains the win-amd64 behavior: There are a couple of places > in mtrand where array indices and sizes are C long instead of npy_intp, > for example in the randint function: > >

Re: [Numpy-discussion] The NumPy Mandelbrot code 16x slower than Fortran

Den 23.01.2012 13:09, skrev Sebastian Haase: > > I would think that interactive zooming would be quite nice > ("illuminating") and for that 13 secs would not be tolerable > Well... it's not at the top of my priority list ... ;-) > Sure, that comes under the 'fast enough' issue. But even

Re: [Numpy-discussion] The NumPy Mandelbrot code 16x slower than Fortran

Den 23.01.2012 10:04, skrev Dag Sverre Seljebotn: > On 01/23/2012 05:35 AM, Jonathan Rocher wrote: >> Hi all, >> >> I was reading this while learning about Pytables in more details and the >> origin of its efficiency. This sounds like a problem where out of core >> computation using pytables would

Re: [Numpy-discussion] Cross-covariance function

Den 20.01.2012 13:39, skrev Pierre Haessig: > I don't see how does your function relates to numpy.cov [1]. Is it an > "extended case" function or is there a difference in the underlying math ? > If X is rank n x p, then np.cov(X, rowvar=False) is equal to S after cX = X - X.mean(axis=0)[np.n

Re: [Numpy-discussion] Strange numpy behaviour (bug?)

Never mind this, it was my own mistake as I expected :-) def __chunk(n,size): x = range(0,n,size) x.append(n) return zip(x[:-1],x[1:]) makes it a lot better :) Sturla Den 18.01.2012 06:26, skrev Sturla Molden: > While "playing" with a point-in-polygon test, I ha

[Numpy-discussion] Strange numpy behaviour (bug?)

it fail (or at least it does on my computer, running Enthought 7.2-1 on Win64). Regards, Sturla Molden def __inpolygon_scalar(x,y,poly): # Source code taken from: # http://paulbourke.net/geometry/insidepoly # http://www.ariel.com.au/a/python-point-int-poly.html n = len(pol

Re: [Numpy-discussion] Calculating density based on distance

Den 14.01.2012 21:52, skrev Thiago Franco de Moraes: > Is there a better and faster way of doing that? Is there something in my > Cython implementation I can do to perform better? > > You need to use a kd-tree to make the computation run in O(n log n) time instead of O(n**2). scipy.spatial.cKDTr

Re: [Numpy-discussion] Improving Python+MPI import performance

Den 13.01.2012 22:42, skrev Sturla Molden: > Den 13.01.2012 22:24, skrev Robert Kern: >> Do these systems have a ramdisk capability? > I assume you have seen this as well :) > > http://www.cs.uoregon.edu/Research/paracomp/papers/iccs11/iccs_paper_final.pdf > This paper

Re: [Numpy-discussion] Improving Python+MPI import performance

Den 13.01.2012 22:24, skrev Robert Kern: > Do these systems have a ramdisk capability? I assume you have seen this as well :) http://www.cs.uoregon.edu/Research/paracomp/papers/iccs11/iccs_paper_final.pdf Sturla ___ NumPy-Discussion mailing list NumP

Re: [Numpy-discussion] Improving Python+MPI import performance

Den 13.01.2012 21:21, skrev Dag Sverre Seljebotn: > Another idea: Given your diagnostics, wouldn't dumping the output of > "find" of every path in sys.path to a single text file work well? It probably would, and would also be less prone to synchronization problems than using an MPI broadcast. Ano

Re: [Numpy-discussion] Improving Python+MPI import performance

Den 13.01.2012 02:13, skrev Asher Langton: > intern try a few of these approaches. It turned out that the problem > wasn't so much the simultaneous module loads, but rather the huge > number of failed open() calls (ENOENT) as the interpreter tries to > find the module files. It sounds like there i

Re: [Numpy-discussion] Rebinning numpy array

Fit a poisson distribution (radioactive decay is a Poisson process), recompute lambda for whatever bin-size you need, and compute the new (estimated) bin counts by maximum likehood. It basically becomes a contrained optimization problem. Sturla Den 13.11.2011 17:04, skrev Johannes Bauer: > Hi gr

Re: [Numpy-discussion] iterate over multiple arrays

Den 12.09.2011 08:52, skrev David Froger: Hy everybody, I'm wondering what is the (best) way to apply the same function to multiple arrays. I tried to experiment a bit with this. Here is from an ipython session: Create some arrays: In [1]: import numpy as np In [2]: a = np.zeros(4) In [3]

Re: [Numpy-discussion] [ANN] Cython 0.15

Den 08.08.2011 11:47, skrev Dag Sverre Seljebotn: > This come up now and again and I believe there's several > half-baked/started solutions out there by Cython users, but nothing > that is standard or that I know is carried out to completion. I.e., > you should ask on the cython-users list. It'd

Re: [Numpy-discussion] [ANN] Cython 0.15

Den 06.08.2011 11:18, skrev Dag Sverre Seljebotn: > We are excited to announce the release of Cython 0.15, which is a huge > step forward in achieving full Python language coverage as well as > many new features, optimizations, and bugfixes. > > This is really great. With Cython progressing like t

Re: [Numpy-discussion] Array vectorization in numpy

Den 20.07.2011 09:35, skrev Carlos Becker: > > In my case, sometimes it is required to process 1k images or more, and > 2x speed improvement in this case means 2 hours of processing vs 4. Can you demonstrate that Matlab is faster than NumPy for this task? Sturla _

Re: [Numpy-discussion] Array vectorization in numpy

Den 19.07.2011 11:05, skrev Carlos Becker: N = 100; tic; for I=1:N k = m - 0.5; end toc / N m = rand(2000,2000); Here, Matlab's JIT compiler can probably hoist the invariant out of the loop, and just do I=N k = m - 0.5 Try thi

Re: [Numpy-discussion] Array vectorization in numpy

Den 20.07.2011 08:49, skrev Carlos Becker: > > The main difference is that Matlab is able to take into account a > pre-allocated array/matrix, probably avoiding the creation of a > temporary and writing the results directly in the pre-allocated array. > > I think this is essential to speed up num

Re: [Numpy-discussion] Array vectorization in numpy

Den 19.07.2011 17:49, skrev Carlos Becker: > > - Matlab: 0.0089 > - Numpy: 0.051 > - Numpy with blitz: 0.0043 > > Now blitz is considerably faster! Anyways, I am concerned about numpy > being much slower, in this case taking 2x the time of the previous > operation. > I guess this is because of th

Re: [Numpy-discussion] Array vectorization in numpy

There is a total lack of vectorization in your code, so you are right about the lack of vectorization. What happens is that you see the result of the Matlab JIT compiler speeding up the loop. With a vectorized array expression, there will hardly be any difference. Sturla Den 19.07.2011 11:

[Numpy-discussion] Adding a linear system type to NumPy?

The problem I am thinking we might try to might fix is that programmers with less numerical competence is unaware that the matrix expression (X**-1) * Y should be written as np.linalg.solve(X,Y) I've seen numerous times matrix expressions being typed exactly as written in linear algeb

Re: [Numpy-discussion] inverting and calculating eigenvalues for many small matrices

Den 11.07.2011 23:01, skrev Daniel Wheeler: > The above uses "map" to fake a vector solution, but this is heinously > slow. Are there any better ways to do this without resorting to cython > or weave (would it even be faster (or possible) to use "np.linalg.eig" > and "np.linalg.inv" within cython)?

Re: [Numpy-discussion] Are .M and .H removed in NumPy 1.6?

Den 12.07.2011 15:55, skrev Charles R Harris: On Tue, Jul 12, 2011 at 7:36 AM, Sturla Molden <mailto:stu...@molden.no>> wrote: After upgrading EPD, I just discovered that my ndarrays no longer have .M and .H attributes. Were they deprectated, or is my NumPy not working

[Numpy-discussion] Are .M and .H removed in NumPy 1.6?

After upgrading EPD, I just discovered that my ndarrays no longer have .M and .H attributes. Were they deprectated, or is my NumPy not working correctly? Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman

Re: [Numpy-discussion] New arrays in 1.6 not always C-contiguous

Den 07.07.2011 14:10, skrev Jens Jørgen Mortensen: > So, this means I can't count on new arrays being C-contiguous any more. > I guess there is a good reason for this. Work with linear algebra (LAPACK) caused excessive and redundant array transpositions. Arrays would be transposed from C to Fortr

Re: [Numpy-discussion] New arrays in 1.6 not always C-contiguous

Den 07.07.2011 15:24, skrev Yoshi Rokuko: > thank you for pointing that out! > > so how do you change your numpy related c code now, would you like to share? > Regardless or memory layout, we can always access element array[i,j,k] like this: const int s0 = array->strides[0]; const int s1 = array

Re: [Numpy-discussion] Current status of 64 bit windows support.

Den 01.07.2011 19:22, skrev Charles R Harris: > Just curious as to what folks know about the current status of the > free windows 64 bit compilers. I know things were dicey with gcc and > gfortran some two years ago, but... well, two years have passed. This Windows 7 SDK is free (as in beer). It

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 16.06.2011 19:55, skrev Sturla Molden: > > The meta-data for the ndarray (strides, shape, dtype, etc.) and the name > of the shared memory segment. As there is no fork, we must tell the > other process where to find the shared memory and what to do with it. > Which by the

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 16.06.2011 19:47, skrev Christopher Barker: > > What do you need to pickle if you're using shared memory? The meta-data for the ndarray (strides, shape, dtype, etc.) and the name of the shared memory segment. As there is no fork, we must tell the other process where to find the shared memory

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 16.06.2011 19:23, skrev Robin: > > If you are on Linux or Mac then fork works nicely so you have read > only shared memory you just have to put it in a module before the fork > (so before pool = Pool() ) and then all the subprocesses can access it > without any pickling required. ie > myutil.da

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 16.06.2011 18:44, skrev Christopher Barker: > > I'll bet this is a big issue, and one I'm curious about how to > address, I have another problem where I need to multi-process, and I'd > love to know a way to pass data to the other process and back > *without* going through pickle. maybe memm

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 16.06.2011 16:16, skrev Bruce Southey: > The idea was to calculate: > innerProduct =numpy.sum(array1*array2) where array1 and array2 are, in > general, multidimensional. If you want to do that in parallel, fast an with minimal overhead, it's a typical OpenMP problem. If the dimensions are unk

Re: [Numpy-discussion] Using numpy.fromfile with structured array skipping some elements.

Den 16.06.2011 15:02, skrev Michael Klitgaard: > Now the data is read in and I can access it, but I have the 'junk' in > the array, which annoys me. > Is there a way to remove the junk data, or skip it with fromfile ? Not that I know of, unless you are ok with a loop (either in Python or C). If i

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 15.06.2011 23:22, skrev Christopher Barker: > > I think the issue got confused -- the OP was not looking to speed up a > matrix multiply, but rather to speed up a whole bunch of independent > matrix multiplies. I would do it like this: 1. Write a Fortran function that make multiple calls DGEM

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 15.06.2011 23:22, skrev Christopher Barker: > > It would also would be great if someone that actually understands this > stuff could look at his code and explain why the slowdown occurs (hint, > hint!) > Not sure I qualify, but I think I notice several potential problems in the OP's multiproc

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Perhaps it is time to write somthing in the SciPy cookbook about parallel computing with NumPy? It seems to be certain problems that are discussed again and again. These are some issues that come to mind (I'm sure there is more): - The difference between I/O bound, memory bound, and CPU bound w

Re: [Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Den 13.06.2011 19:51, skrev srean: > If you are on an intel machine and you have MKL libraries around I > would strongly recommend that you use the matrix multiplication > routine if possible. MKL will do the parallelization for you. Well, > any good BLAS implementation would do the same, you dont

Re: [Numpy-discussion] need help building a numpy extension that uses fftpack on windows

Den 02.06.2011 21:07, skrev Ralf Gommers: > > After search for multi-taper I found > http://projects.scipy.org/scipy/ticket/608. Too bad no one has gotten > around to review your contribution before, it sounds like a good > addition for the scipy.signal module. I once implemented DPSS tapers in

Re: [Numpy-discussion] Mapping of dtype to C types

Den 09.05.2011 15:58, skrev Keith Goodman: > On Mon, May 9, 2011 at 1:46 AM, Pauli Virtanen wrote: >> Sun, 08 May 2011 14:45:45 -0700, Keith Goodman wrote: >>> I'm writing a function that accepts four possible dtypes: int32, int64, >>> float32, float64. The function will call a C extension (wrappe

Re: [Numpy-discussion] Python crashes while printing a huge numpy array (Windows 1.6b2)

Den 23.04.2011 16:09, skrev josef.p...@gmail.com: > to rule out two more cases > > numpy 1.5.1 on python 2.7.1 32 bit : ValueError > numpy 1.5.1 on python 3.264 bit: MemoryError > numpy 1.5.1 on Python 2.7.1 64 bit: MemoryError Sturla ___ NumPy-

Re: [Numpy-discussion] Shared memory ndarrays (update)

Den 11.04.2011 21:15, skrev srean: Got you and thanks a lot for the explanation. I am not using Queues so I think I am safe for the time being. Given that you have worked a lot on these issues, would you recommend plain mmapped numpy arrays over |multiprocessing.Array| | With multiprocessing

Re: [Numpy-discussion] Shared memory ndarrays (update)

Den 11.04.2011 01:20, skrev Sturla Molden: > > Changes: > > - 64-bit support. > - Memory leak on Linux/Unix should be gone (monkey patch for os._exit). > - Added a global lock as there are callbacks to Python (the GIL is not > sufficient serialization). I will also add a ba

Re: [Numpy-discussion] Shared memory ndarrays (update)

Den 11.04.2011 14:58, skrev Zbigniew Jędrzejewski-Szmek: > Hi, > it could, but you'd have to do the parsing of data yourself. So nothing > fancy unless you want to reimplement numpy in Java :) Not really. Only the data buffer is stored in shared memory. If you can pass the required fields to Java

Re: [Numpy-discussion] Shared memory ndarrays (update)

"Shared memory" is memory mapping from the paging file (i.e. RAM), not a file on disk. They can have a name or be anonymous. I have explained why we need named shared memory before. If you didn't understand it, try to pass an instance of |multiprocessing.Array over | |multiprocessing.Queue. |S

Re: [Numpy-discussion] Classes as records within an numpy array

Den 11.04.2011 04:17, skrev Sturla Molden: > > Consider that x[i][j] means ( x[i] )[j]. Then consider that x[i] in your > case returns an instance of Record, not ndarray. Sorry, disregard this. a[i] is the same as a[i,:], which means that a[i,j] and a[i][j] will return the same.

<    1   2   3   4   5   6   7   8   9   >