Re: [Numpy-discussion] is there an efficient way to get a random set of subsets/combinations?

[Val Kalatsky]

Hi Slava,

Since your k is only 10, here is a quickie:

import numpy as np
arr = np.arange(n)
for i in range(k):
np.random.shuffle(arr)
print np.sort(arr[:p])

If your ever get non-unique entries in a set of k=10 for your n and p,
consider yourself lucky:)
Val

On Mon, Feb 20, 2012 at 10:35 PM, Yaroslav Halchenko
wrote:

> Hi to all Numeric  Python experts,
>
> could not think of a mailing list with better fit to my question which
> might
> have an obvious answer:
>
> straightforward (naive) Python code to answer my question would be
> something like
>
> import random, itertools
> n,p,k=100,50,10  # don't try to run with this numbers! ;)
> print random.sample(list(itertools.combinations(range(n), p)), k)
>
> so the goal is to get k (non-repeating) p-subsets of n, where n and p
> prohibitively large to first populate the full set of combinations.
>
> Thank you in advance ;-)
> --
> =--=
> Keep in touch www.onerussian.com
> Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] is there an efficient way to get a random set of subsets/combinations?

[Christopher Jordan-Squire]

If you're using numpy 2.0 (the development branch), the function
numpy.random.choice might do what you're looking for.

-Chris

On Mon, Feb 20, 2012 at 8:35 PM, Yaroslav Halchenko
 wrote:
> Hi to all Numeric  Python experts,
>
> could not think of a mailing list with better fit to my question which might
> have an obvious answer:
>
> straightforward (naive) Python code to answer my question would be
> something like
>
> import random, itertools
> n,p,k=100,50,10  # don't try to run with this numbers! ;)
> print random.sample(list(itertools.combinations(range(n), p)), k)
>
> so the goal is to get k (non-repeating) p-subsets of n, where n and p
> prohibitively large to first populate the full set of combinations.
>
> Thank you in advance ;-)
> --
> =--=
> Keep in touch                                     www.onerussian.com
> Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] is there an efficient way to get a random set of subsets/combinations?

[Yaroslav Halchenko]

Hi to all Numeric  Python experts,

could not think of a mailing list with better fit to my question which might
have an obvious answer:

straightforward (naive) Python code to answer my question would be
something like

import random, itertools
n,p,k=100,50,10  # don't try to run with this numbers! ;)
print random.sample(list(itertools.combinations(range(n), p)), k)

so the goal is to get k (non-repeating) p-subsets of n, where n and p
prohibitively large to first populate the full set of combinations.

Thank you in advance ;-)
-- 
=--=
Keep in touch www.onerussian.com
Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Travis Oliphant]

Interesting you bring this up.   I actually have a working prototype of using 
Python to emit LLVM.   I will be showing it at the HPC tutorial that I am 
giving at PyCon.I will be making this available after PyCon to a wider 
audience as open source.  

It uses llvm-py (modified to work with LLVM 3.0) and code I wrote to do the 
translation from Python byte-code to LLVM.   This LLVM can then be "JIT"ed.   I 
have several applications that I would like to use this for.   It would be 
possible to write "more of NumPy" using this approach. Initially, it makes 
it *very* easy to create a machine-code ufunc from Python code.   There are 
other use-cases of having loops written in Python and plugged in to a 
calculation, filtering, or indexing framework that this system will be useful 
for.  

There is still a need for a core data-type object, a core array object, and a 
core calculation object.   Maybe some-day these cores can be shrunk to a 
smaller subset and more of something along the lines of LLVM generation from 
Python can be used.   But, there is a lot of work to do before that is 
possible.But, a lot of the currently pre-compiled loops can be done on the 
fly instead using this approach.There are several things I'm working on in 
that direction. 

This is not PyPy.   It certainly uses the same ideas that they are using, but 
instead it fits into the CPython run-time and doesn't require changing the 
whole ecosystem. If you are interested in this work let me know.  I think 
I'm going to call the project numpy-llvm, or fast-py, or something like that.   
It is available on github and will be open source (but it's still under active 
development). 

Here is an example of the code to create a ufunc using the system (this is like 
vectorize, but it creates machine code and by-passes the interpreter and so is 
100x faster).  

from math import sin, pi

def sinc(x):
if x==0:
return 1.0
else:
return sin(x*pi)/(pi*x)

from translate import Translate
t = Translate(sinc)
t.translate()
print t.mod

res = t.make_ufunc('sinc')


-Travis



On Feb 20, 2012, at 10:55 AM, Sturla Molden wrote:

> Den 20.02.2012 17:42, skrev Sturla Molden:
>> There are still other options than C or C++ that are worth considering.
>> One would be to write NumPy in Python. E.g. we could use LLVM as a
>> JIT-compiler and produce the performance critical code we need on the fly.
>> 
>> 
> 
> LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster 
> than GCC and often produces better machine code. They can therefore be 
> used inside an array library. It would give a faster NumPy, and we could 
> keep most of it in Python.
> 
> Sturla
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Robert Kern]

On Mon, Feb 20, 2012 at 19:55, Paul Anton Letnes
 wrote:
>
> On 20. feb. 2012, at 16:29, Sturla Molden wrote:

>>> - in newer standards it has some nontrivial mathematical functions: gamma, 
>>> bessel, etc. that numpy lacks right now
>>
>> That belongs to SciPy.
>
> I don't see exactly why. Why should numpy have exponential but not gamma 
> functions? The division seems kinda arbitrary. Not that I am arguing 
> violently for bessel functions in numpy.

The semi-arbitrary dividing line that we have settled on is C99. If a
special function is in the C99 standard, we'll accept an
implementation for it in numpy. Part (well, most) of the rationale is
just to have a clear dividing line even if it's fairly arbitrary. The
other part is that if a decidedly non-mathematically-focused standard
like C99 includes a special function in its standard library, then
odds are good that it's something that is widely used enough as a
building block for other things.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Sturla Molden]

Den 20.02.2012 21:12, skrev Sturla Molden:
>
> If you need to control the lifetime of an object, make an inner block
> with curly brackets, and declare it on top of the block. Don't call new
> and delete to control where you want it to be allocated and deallocated.
> Nothing goes on the heap unless STL puts it there.
>

Here is an example:


// bad

Foo *bar = new Foo();

delete Foo;


// ok
{
 Foo bar();

}

Remember C++ does not allow a "finally" clause to exception handling. 
You cannot do this:

try {
 Foo *bar = new Foo();
} finally { // syntax error
 delete Foo;
}

So...

try {
 Foo *bar = new Foo();
} catch(...) {

}
// might not get here, possible
// resource leak
delete Foo;


Which is why we should always do this:

{
 Foo bar();

}

This is perhaps the most common source of errors in C++ code. If we use 
C++ in the NumPy core, we need a Nazi regime against these type of 
obscure errors.


Sturla









___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray and lazy evaluation

[James Bergstra]

On Mon, Feb 20, 2012 at 2:57 PM, Lluís  wrote:

> James Bergstra writes:
> [...]
> > I should add that the biggest benefit of expressing things as compound
> > expressions in this way is not in saving temporaries (though that is
> nice) it's
> > being able to express enough computation work at a time that it offsets
> the time
> > required to ship the arguments off to a GPU for evaluation!
>
> Right, that's exacly what you need for an "external computation" to pay
> off.
>
> Just out of curiosity (feel free to respond with a RTFM or a RTFP :)), do
> you
> support any of these? (sorry for the made-up names)
>
> * automatic transfer double-buffering
>

Not currently, but it would be quite straightforward to do it. Email
theano-dev and ask how if you really want to know.


>
> * automatic problem partitioning into domains (e.g., multiple GPUs; even
> better
>  if also supports nodes - MPI -)
>

Not currently, and it would be hard.


>
> * point-specific computations (e.g., code dependant on the thread id,
> although
>  this can also be expressed in other ways, like index ranges)
>
>
No.


> * point-relative computations (the most common would be a stencil)
>
>
No, but I think theano provides a decent expression language to tackle
this. The "Composite" element-wise code generator is an example of how I
would think about this. It provides point-relative computations across
several arguments.  You might want something different that applies a
stencil computation across one or several arguments... the "scan" operator
was another foray into this territory, and it got tricky when the stencil
operation could have side-effects (like random number generation) and could
define it's own input domain (stencil shape), but the result is quite
powerful.

-- 
http://www-etud.iro.umontreal.ca/~bergstrj
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Sturla Molden]

Den 20.02.2012 20:14, skrev Daniele Nicolodi:

> Hello Sturla, unrelated to the numpy tewrite debate, can you please 
> suggest some resources you think can be used to learn how to program 
> C++ "the proper way"? Thank you. Cheers, 

This is totally OT on this list, however ...

Scott Meyer's books have been mentioned. Also look at some litterature 
on the STL (e.g. Josuittis). Getting the Boost library is essential as 
well. The Qt library have many examples of beautiful C++.

But the most important part, in my opinion, is to put the "C with 
classes" mentality away. Look at it as compiled Python or Java. The STL 
(the standard C++ library) has classes that do the same as the types we 
use in Python  --- there are parallels to tuple, dict, set, list, deque, 
etc. The STL is actually richer than Python. Just use them the way we 
use Python. With C++11 (the latest standard), even for loops can be like 
Python. There are lamdas and closures, to be used as in Python, and 
there is an 'auto' keyword for type inference; you don't have to declare 
the type of a variable, the compiler will figure it out.

Don't use new[] just because you can, when there is std::vector that 
behaves lika Python list.

If you need to allocate a resource, wrap it in a class. Allocate from 
the contructor and deallocate from the destructor. That way an exception 
cannot cause a resource leak, and the clean-up code will be called 
automatically when the object fall of the stack.

If you need to control the lifetime of an object, make an inner block 
with curly brackets, and declare it on top of the block. Don't call new 
and delete to control where you want it to be allocated and deallocated. 
Nothing goes on the heap unless STL puts it there.

Always put objects on the stack, never allocate to a pointer with new.  
Always use references, and forget about pointers. This has to do with 
putting the "C with classes" mentality away. Always implement a copy 
constructor so the classes work with the STL.

std:: vector  x(n); // ok
void foobar(std:: vector&  x); // ok

double* x = new double [n]; // bad
std:: vector *x = new std:: vector (n); // bad
void foobar(std:: vector*  x); // bad

If you get any textbook on Windows programming from Microsoft Press, you 
have an excellent resource on what not to do. Verbose functions and 
field names, Hungarian notation, factories instead of constructors, 
etc.  If you find yourself using macros or template magic to avoid the 
overhead of a virtual function (MFC, ATL, wxWidgets, FOX), for the 
expense of readability, you are probably doing something you shouldn't. 
COM is probably the worst example I know of, just compare the beautiful 
OpenGL to Direct3D. VTK is another example of what I consider ugly C++.

But that's just my opinion.


Sturla
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[James Bergstra]

Looks like Dag forked the discussion of lazy evaluation to a new thread
 ([Numpy-discussion] ndarray and lazy evaluation).

There are actually several projects inspired by this sort of design: off
the top of my head I can think of Theano, copperhead, numexpr, arguably
sympy, and some non-public code by Nicolas Pinto. So I think the strengths
of the approach in principle are established... the big question is how to
make this approach easy to use in all the settings where it could be
useful. I don't think any of these projects has gotten that totally right.

-JB

On Mon, Feb 20, 2012 at 2:41 PM, Lluís  wrote:

> Lluís  writes:
>
> > Francesc Alted writes:
> >> On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote:
> >>> You need at least a slightly different Python API to get anywhere, so
> >>> numexpr/Theano is the right place to work on an implementation of this
> >>> idea. Of course it would be nice if numexpr/Theano offered something as
> >>> convenient as
> >>>
> >>> with lazy:
> >>> arr = A + B + C # with all of these NumPy arrays
> >>> # compute upon exiting…
>
> >> Hmm, that would be cute indeed.  Do you have an idea on how the code in
> the with
> >> context could be passed to the Python AST compiler (à la
> numexpr.evaluate("A + B
> >> + C"))?
>
> > Well, I started writing some experiments to "almost transparently"
> translate
> > regular ndarray operations to numexpr strings (or others) using only
> python
> > code.
> [...]
> > My target was to use this to also generate optimized GPU kernels
> in-flight using
> > pycuda, but I think some other relatively recent project already
> performed
> > something similar (w.r.t. generating cuda kernels out of python
> expressions).
>
> Aaahhh, I just had a quick look at Theano and it seems it's the project I
> was
> referring to.
>
> Good job! :)
>
>
> Lluis
>
> --
>  "And it's much the same thing with knowledge, for whenever you learn
>  something new, the whole world becomes that much richer."
>  -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
>  Tollbooth
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
http://www-etud.iro.umontreal.ca/~bergstrj
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray and lazy evaluation

[Lluís]

James Bergstra writes:
[...]
> I should add that the biggest benefit of expressing things as compound
> expressions in this way is not in saving temporaries (though that is nice) 
> it's
> being able to express enough computation work at a time that it offsets the 
> time
> required to ship the arguments off to a GPU for evaluation!

Right, that's exacly what you need for an "external computation" to pay off.

Just out of curiosity (feel free to respond with a RTFM or a RTFP :)), do you
support any of these? (sorry for the made-up names)

* automatic transfer double-buffering

* automatic problem partitioning into domains (e.g., multiple GPUs; even better
  if also supports nodes - MPI -)

* point-specific computations (e.g., code dependant on the thread id, although
  this can also be expressed in other ways, like index ranges)

* point-relative computations (the most common would be a stencil)

If you have all of them, then I'd say the project has a huge potential for total
world dominance :)


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Paul Anton Letnes]

On 20. feb. 2012, at 16:29, Sturla Molden wrote:

> Den 20.02.2012 08:35, skrev Paul Anton Letnes:
>> In the language wars, I have one question. Why is Fortran not being 
>> considered? Fortran already implements many of the features that we want in 
>> NumPy:
> 
> Yes ... but it does not make Fortran a systems programming language. 
> Making NumPy is different from using it.
> 
>> - slicing and similar operations, at least some of the fancy indexing kind
>> - element-wise array operations and function calls
>> - array bounds-checking and other debugging aid (with debugging flags)
> 
> That is nice for numerical computing, but not really needed to make NumPy.
> 
> 
>> - arrays that mentally map very well onto numpy arrays. To me, this spells 
>> +1 to ease of contribution, over some abstract C/C++ template
> 
> Mentally perhaps, but not binary. NumPy needs uniformly strided memory 
> on the binary level. Fortran just gives this at the mental level. E.g. 
> there is nothing that dictates a Fortran pointer has to be a view, the 
> compiler is free to employ copy-in copy-out. In Fortran, a function call 
> can invalidate a pointer.  One would therefore have to store the array 
> in an array of integer*1, and use the intrinsic function transfer() to 
> parse the contents into NumPy dtypes.
> 
>> - in newer standards it has some nontrivial mathematical functions: gamma, 
>> bessel, etc. that numpy lacks right now
> 
> That belongs to SciPy.

I don't see exactly why. Why should numpy have exponential but not gamma 
functions? The division seems kinda arbitrary. Not that I am arguing violently 
for bessel functions in numpy.

>> - compilers that are good at optimizing for floating-point performance, 
>> because that's what Fortran is all about
> 
> Insanely good, but not when we start to do the (binary, not mentally) 
> strided access that NumPy needs. (Not that C compilers would be any better.)
> 
> 
> 
>> - not Fortran as such, but BLAS and LAPACK are easily accessed by Fortran
>> - possibly other numerical libraries that can be helpful
>> - Fortran has, in its newer standards, thought of C interoperability. We 
>> could still keep bits of the code in C (or even C++?) if we'd like to, or 
>> perhaps f2py/Cython could do the wrapping.
> 
> Not f2py, as it depends on NumPy.
> 
> - some programmers know Fortran better than C++. Fortran is at least used by 
> many science guys, like me.
> 
> 
> That is a valid arguments. Fortran is also much easier to read and debug.
> 
> 
> Sturla

Thanks for an excellent answer, Sturla - very informative indeed.

Paul.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Lluís]

Lluís  writes:

> Francesc Alted writes:
>> On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote:
>>> You need at least a slightly different Python API to get anywhere, so 
>>> numexpr/Theano is the right place to work on an implementation of this 
>>> idea. Of course it would be nice if numexpr/Theano offered something as 
>>> convenient as
>>> 
>>> with lazy:
>>> arr = A + B + C # with all of these NumPy arrays
>>> # compute upon exiting…

>> Hmm, that would be cute indeed.  Do you have an idea on how the code in the 
>> with
>> context could be passed to the Python AST compiler (à la numexpr.evaluate("A 
>> + B
>> + C"))?

> Well, I started writing some experiments to "almost transparently" translate
> regular ndarray operations to numexpr strings (or others) using only python
> code.
[...]
> My target was to use this to also generate optimized GPU kernels in-flight 
> using
> pycuda, but I think some other relatively recent project already performed
> something similar (w.r.t. generating cuda kernels out of python expressions).

Aaahhh, I just had a quick look at Theano and it seems it's the project I was
referring to.

Good job! :)


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Neal Becker]

Charles R Harris wrote:

> On Fri, Feb 17, 2012 at 12:09 PM, Benjamin Root  wrote:
> 
>>
>>
>> On Fri, Feb 17, 2012 at 1:00 PM, Christopher Jordan-Squire <
>> cjord...@uw.edu> wrote:
>>
>>> On Fri, Feb 17, 2012 at 10:21 AM, Mark Wiebe  wrote:
>>> > On Fri, Feb 17, 2012 at 11:52 AM, Eric Firing 
>>> wrote:
>>> >>
>>> >> On 02/17/2012 05:39 AM, Charles R Harris wrote:
>>> >> >
>>> >> >
>>> >> > On Fri, Feb 17, 2012 at 8:01 AM, David Cournapeau <
>>> courn...@gmail.com
>>> >> > > wrote:
>>> >> >
>>> >> > Hi Travis,
>>> >> >
>>> >> > On Thu, Feb 16, 2012 at 10:39 PM, Travis Oliphant
>>> >> > mailto:tra...@continuum.io>> wrote:
>>> >> >  > Mark Wiebe and I have been discussing off and on (as well as
>>> >> > talking with Charles) a good way forward to balance two competing
>>> >> > desires:
>>> >> >  >
>>> >> >  >* addition of new features that are needed in NumPy
>>> >> >  >* improving the code-base generally and moving towards
>>> a
>>> >> > more maintainable NumPy
>>> >> >  >
>>> >> >  > I know there are load voices for just focusing on the second
>>> of
>>> >> > these and avoiding the first until we have finished that.  I
>>> >> > recognize the need to improve the code base, but I will also be
>>> >> > pushing for improvements to the feature-set and user experience
>>> in
>>> >> > the process.
>>> >> >  >
>>> >> >  > As a result, I am proposing a rough outline for releases over
>>> the
>>> >> > next year:
>>> >> >  >
>>> >> >  >* NumPy 1.7 to come out as soon as the serious bugs
>>> can be
>>> >> > eliminated.  Bryan, Francesc, Mark, and I are able to help triage
>>> >> > some of those.
>>> >> >  >
>>> >> >  >* NumPy 1.8 to come out in July which will have as many
>>> >> > ABI-compatible feature enhancements as we can add while improving
>>> >> > test coverage and code cleanup.   I will post to this list more
>>> >> > details of what we plan to address with it later.Included for
>>> >> > possible inclusion are:
>>> >> >  >* resolving the NA/missing-data issues
>>> >> >  >* finishing group-by
>>> >> >  >* incorporating the start of label arrays
>>> >> >  >* incorporating a meta-object
>>> >> >  >* a few new dtypes (variable-length string,
>>> >> > varialbe-length unicode and an enum type)
>>> >> >  >* adding ufunc support for flexible dtypes and possibly
>>> >> > structured arrays
>>> >> >  >* allowing generalized ufuncs to work on more kinds of
>>> >> > arrays besides just contiguous
>>> >> >  >* improving the ability for NumPy to receive
>>> JIT-generated
>>> >> > function pointers for ufuncs and other calculation opportunities
>>> >> >  >* adding "filters" to Input and Output
>>> >> >  >* simple computed fields for dtypes
>>> >> >  >* accepting a Data-Type specification as a class or
>>> JSON
>>> >> > file
>>> >> >  >* work towards improving the dtype-addition mechanism
>>> >> >  >* re-factoring of code so that it can compile with a
>>> C++
>>> >> > compiler and be minimally dependent on Python data-structures.
>>> >> >
>>> >> > This is a pretty exciting list of features. What is the rationale
>>> >> > for
>>> >> > code being compiled as C++ ? IMO, it will be difficult to do so
>>> >> > without preventing useful C constructs, and without removing
>>> some of
>>> >> > the existing features (like our use of C99 complex). The subset
>>> that
>>> >> > is both C and C++ compatible is quite constraining.
>>> >> >
>>> >> >
>>> >> > I'm in favor of this myself, C++ would allow a lot code cleanup and
>>> make
>>> >> > it easier to provide an extensible base, I think it would be a
>>> natural
>>> >> > fit with numpy. Of course, some C++ projects become tangled messes of
>>> >> > inheritance, but I'd be very interested in seeing what a good C++
>>> >> > designer like Mark, intimately familiar with the numpy code base,
>>> could
>>> >> > do. This opportunity might not come by again anytime soon and I
>>> think we
>>> >> > should grab onto it. The initial step would be a release whose code
>>> that
>>> >> > would compile in both C/C++, which mostly comes down to removing C++
>>> >> > keywords like 'new'.
>>> >> >
>>> >> > I did suggest running it by you for build issues, so please raise any
>>> >> > you can think of. Note that MatPlotLib is in C++, so I don't think
>>> the
>>> >> > problems are insurmountable. And choosing a set of compilers to
>>> support
>>> >> > is something that will need to be done.
>>> >>
>>> >> It's true that matplotlib relies heavily on C++, both via the Agg
>>> >> library and in its own extension code.  Personally, I don't like this;
>>> I
>>> >> think it raises the barrier to contributing.  C++ is an order of
>>> >> magnitu

Re: [Numpy-discussion] Proposed Roadmap Overview

[Lluís]

Francesc Alted writes:

> On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote:
>> You need at least a slightly different Python API to get anywhere, so 
>> numexpr/Theano is the right place to work on an implementation of this 
>> idea. Of course it would be nice if numexpr/Theano offered something as 
>> convenient as
>> 
>> with lazy:
>> arr = A + B + C # with all of these NumPy arrays
>> # compute upon exiting…

> Hmm, that would be cute indeed.  Do you have an idea on how the code in the 
> with
> context could be passed to the Python AST compiler (à la numexpr.evaluate("A 
> + B
> + C"))?

Well, I started writing some experiments to "almost transparently" translate
regular ndarray operations to numexpr strings (or others) using only python
code.

The concept is very simple:

# you only need the first one to start building the AST
a = lazy(np.arange(16))
b = np.arange(16)
res = a + b + 3
print evaluate(res)
# the actual evaluation can be delayed to something like __repr__ or __str__
print repr(res)
print res
# you could also delay evaluation until someone uses res to create a new 
array

My target was to use this to also generate optimized GPU kernels in-flight using
pycuda, but I think some other relatively recent project already performed
something similar (w.r.t. generating cuda kernels out of python expressions).

The supporting code for numexpr was something like:

import numexpr
import numpy as np

def build_arg_expr (arg, args):
if isinstance(arg, Expr):
   # recursively build the expression
   arg_expr, arg_args = arg.build_expr()
   args.update(arg_args)
   return arg_expr
else:
   # unique argument identifier
   arg_id = "arg_%d" % id(arg)
   args[arg_id] = arg
   return arg_id
 
# generic expression builder
class Expr:
  def evaluate(self):
  expr, args = self.build_expr()
  return numexpr.evaluate(expr, local_dict = args, global_dict = {})
 
  def __repr__ (self):
  return self.evaluate().__repr__()
 
  def __str__ (self):
  return self.evaluate().__str__()
 
  def __add__ (self, other):
  return ExprAdd(self, other)
 
# expression builder for adds
class ExprAdd(Expr):
  def __init__(self, arg1, arg2):
  self.arg1 = arg1
  self.arg2 = arg2
 
  def build_expr(self):
  args = {}
  expr1 = build_arg_expr(self.arg1, args)
  expr2 = build_arg_expr(self.arg2, args)
  return "("+expr1+") + ("+expr2+")", args
 
# ndarray-like class to generate expression builders
class LazyNdArray(np.ndarray):
  def __add__ (self, other):
  return ExprAdd(self, other)
 
# build a LazyNdArray
def lazy (arg):
return arg.view(LazyNdArray)
 
# evaluate with numexpr an arbitrary expression builder 
def evaluate(arg):
return arg.evaluate()


The thing here is to always return to the user something that looks like an
ndarray.

As you can see the whole thing is not very complex, but some less funny code had
to be written meanwhile for work and I just dropped this :)


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray and lazy evaluation

[James Bergstra]

On Mon, Feb 20, 2012 at 1:01 PM, James Bergstra wrote:

> On Mon, Feb 20, 2012 at 12:28 PM, Francesc Alted wrote:
>
>> On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote:
>> > You need at least a slightly different Python API to get anywhere, so
>> > numexpr/Theano is the right place to work on an implementation of this
>> > idea. Of course it would be nice if numexpr/Theano offered something as
>> > convenient as
>> >
>> > with lazy:
>> > arr = A + B + C # with all of these NumPy arrays
>> > # compute upon exiting…
>>
>> Hmm, that would be cute indeed.  Do you have an idea on how the code in
>> the with context could be passed to the Python AST compiler (à la
>> numexpr.evaluate("A + B + C"))?
>>
>>
> The biggest problem with the numexpr approach (e.g. evaluate("A + B + C"))
> whether the programmer has to type the quotes or not, is that the
> sub-program has to be completely expressed in the sub-language.
>
> If I write
>
> >>> def f(x): return x[:3]
> >>> numexpr.evaluate("A + B + f(C)")
>
> I would like that to be fast, but it's not obvious at all how that would
> work. We would be asking numexpr to introspect arbitrary callable python
> objects, and recompile arbitrary Python code, effectively setting up the
> expectation in the user's mind that numexpr is re-implementing an entire
> compiler. That can be fast obviously, but it seems to me to represent
> significant departure from numpy's focus, which I always thought was the
> data-container rather than the expression evaluation (though maybe this
> firestorm of discussion is aimed at changing this?)
>
> Theano went with another option which was to replace the A, B, and C
> variables with objects that have a modified __add__. Theano's back-end can
> be slow at times and the codebase can feel like a heavy dependency, but my
> feeling is still that this is a great approach to getting really fast
> implementations of compound expressions.
>
> The context syntax you suggest using is a little ambiguous in that the
> indented block of a with statement block includes *statements* whereas what
> you mean to build in the indented block is a *single expression* graph.
>  You could maybe get the right effect with something like
>
> A, B, C = np.random.rand(3, 5)
>
> expr = np.compound_expression()
> with np.expression_builder(expr) as foo:
>arr = A + B + C
>brr = A + B * C
>foo.return((arr, brr))
>
> # compute arr and brr as quickly as possible
> a, b = expr.run()
>
> # modify one of the arrays that the expression was compiled to use
> A[:] += 1
>
> # re-run the compiled expression on the new value
> a, b = expr.run()
>
> - JB
>

I should add that the biggest benefit of expressing things as compound
expressions in this way is not in saving temporaries (though that is nice)
it's being able to express enough computation work at a time that it
offsets the time required to ship the arguments off to a GPU for
evaluation!  This has been a *huge* win reaped by the Theano approach, it
works really well.  The abstraction boundary offered by this sort of
expression graph has been really effective.

This speaks even more to the importance of distinguishing between the data
container (e.g. numpy, Theano's internal ones, PyOpenCL's one, PyCUDA's
one) and the expression compilation and evaluation infrastructures (e.g.
Theano, numexpr, cython).  The goal should be as much as possible to
separate these two so that programs can be expressed in a natural way, and
then evaluated using containers that are suited to the program.

- JB

-- 
James Bergstra, Ph.D.
Research Scientist
Rowland Institute, Harvard University
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray and lazy evaluation

[James Bergstra]

On Mon, Feb 20, 2012 at 12:28 PM, Francesc Alted wrote:

> On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote:
> > You need at least a slightly different Python API to get anywhere, so
> > numexpr/Theano is the right place to work on an implementation of this
> > idea. Of course it would be nice if numexpr/Theano offered something as
> > convenient as
> >
> > with lazy:
> > arr = A + B + C # with all of these NumPy arrays
> > # compute upon exiting…
>
> Hmm, that would be cute indeed.  Do you have an idea on how the code in
> the with context could be passed to the Python AST compiler (à la
> numexpr.evaluate("A + B + C"))?
>
>
The biggest problem with the numexpr approach (e.g. evaluate("A + B + C"))
whether the programmer has to type the quotes or not, is that the
sub-program has to be completely expressed in the sub-language.

If I write

>>> def f(x): return x[:3]
>>> numexpr.evaluate("A + B + f(C)")

I would like that to be fast, but it's not obvious at all how that would
work. We would be asking numexpr to introspect arbitrary callable python
objects, and recompile arbitrary Python code, effectively setting up the
expectation in the user's mind that numexpr is re-implementing an entire
compiler. That can be fast obviously, but it seems to me to represent
significant departure from numpy's focus, which I always thought was the
data-container rather than the expression evaluation (though maybe this
firestorm of discussion is aimed at changing this?)

Theano went with another option which was to replace the A, B, and C
variables with objects that have a modified __add__. Theano's back-end can
be slow at times and the codebase can feel like a heavy dependency, but my
feeling is still that this is a great approach to getting really fast
implementations of compound expressions.

The context syntax you suggest using is a little ambiguous in that the
indented block of a with statement block includes *statements* whereas what
you mean to build in the indented block is a *single expression* graph.
 You could maybe get the right effect with something like

A, B, C = np.random.rand(3, 5)

expr = np.compound_expression()
with np.expression_builder(expr) as foo:
   arr = A + B + C
   brr = A + B * C
   foo.return((arr, brr))

# compute arr and brr as quickly as possible
a, b = expr.run()

# modify one of the arrays that the expression was compiled to use
A[:] += 1

# re-run the compiled expression on the new value
a, b = expr.run()

- JB

-- 
James Bergstra, Ph.D.
Research Scientist
Rowland Institute, Harvard University
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Matthieu Brucher]

2012/2/20 Daniele Nicolodi 

> On 18/02/12 04:54, Sturla Molden wrote:
> > This is not true. C++ can be much easier, particularly for those who
> > already know Python. The problem: C++ textbooks teach C++ as a subset
> > of C. Writing C in C++ just adds the complexity of C++ on top of C,
> > for no good reason. I can write FORTRAN in any language, it does not
> > mean it is a good idea. We would have to start by teaching people to
> > write good C++.  E.g., always use the STL like Python built-in types
> > if possible. Dynamic memory should be std::vector, not new or malloc.
> > Pointers should be replaced with references. We would have to write a
> > C++ programming tutorial that is based on Pyton knowledge instead of
> > C knowledge.
>
> Hello Sturla,
>
> unrelated to the numpy tewrite debate, can you please suggest some
> resources you think can be used to learn how to program C++ "the proper
> way"?
>

One of the best books may be "Accelerated C++" or the new Stroutrup's book
(not the C++ language)

Matthieu
-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Daniele Nicolodi]

On 18/02/12 04:54, Sturla Molden wrote:
> This is not true. C++ can be much easier, particularly for those who
> already know Python. The problem: C++ textbooks teach C++ as a subset
> of C. Writing C in C++ just adds the complexity of C++ on top of C,
> for no good reason. I can write FORTRAN in any language, it does not
> mean it is a good idea. We would have to start by teaching people to
> write good C++.  E.g., always use the STL like Python built-in types
> if possible. Dynamic memory should be std::vector, not new or malloc.
> Pointers should be replaced with references. We would have to write a
> C++ programming tutorial that is based on Pyton knowledge instead of
> C knowledge.

Hello Sturla,

unrelated to the numpy tewrite debate, can you please suggest some
resources you think can be used to learn how to program C++ "the proper
way"?

Thank you. Cheers,
-- 
Daniele
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Forbidden charcter in the "names" argument of genfromtxt?

[Adam Hughes]

Thanks for clearing that up.

On Mon, Feb 20, 2012 at 1:58 PM, Skipper Seabold wrote:

> On Mon, Feb 20, 2012 at 1:35 PM, Brett Olsen 
> wrote:
> > On Sat, Feb 18, 2012 at 8:12 PM, Adam Hughes 
> wrote:
> >> Hey everyone,
> >>
> >> I have timeseries data in which the column label is simply a filename
> from
> >> which the original data was taken.  Here's some sample data:
> >>
> >> name1.txt  name2.txt  name3.txt
> >> 32  34953
> >> 32  03402
> >>
> >> I've noticed that the standard genfromtxt() method works great;
> however, the
> >> names aren't written correctly.  That is, if I use the command:
> >>
> >> print data['name1.txt']
> >>
> >> Nothing happens.
> >>
> >> However, when I remove the file extension, Eg:
> >>
> >> name1  name2  name3
> >> 32  34953
> >> 32  03402
> >>
> >> Then print data['name1'] return (32, 32) as expected.  It seems that the
> >> period in the name isn't compatible with the genfromtxt() names
> attribute.
> >> Is there a workaround, or do I need to restructure my program to get the
> >> extension removed?  I'd rather not do this if possible for reasons that
> >> aren't important for the discussion at hand.
> >
> > It looks like the period is just getting stripped out of the names:
> >
> > In [1]: import numpy as N
> >
> > In [2]: N.genfromtxt('sample.txt', names=True)
> > Out[2]:
> > array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
> >  dtype=[('name1txt', ' ' >
> > Interestingly, this still happens if you supply the names manually:
> >
> > In [17]: def reader(filename):
> >   : infile = open(filename, 'r')
> >   : names = infile.readline().split()
> >   : data = N.genfromtxt(infile, names=names)
> >   : infile.close()
> >   : return data
> >   :
> >
> > In [20]: data = reader('sample.txt')
> >
> > In [21]: data
> > Out[21]:
> > array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
> >  dtype=[('name1txt', ' ' >
> > What you can do is reset the names after genfromtxt is through with it,
> though:
> >
> > In [34]: def reader(filename):
> >   : infile = open(filename, 'r')
> >   : names = infile.readline().split()
> >   : infile.close()
> >   : data = N.genfromtxt(filename, names=True)
> >   : data.dtype.names = names
> >   : return data
> >   :
> >
> > In [35]: data = reader('sample.txt')
> >
> > In [36]: data
> > Out[36]:
> > array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
> >  dtype=[('name1.txt', ' ' >
> > Be warned, I don't know why the period is getting stripped; there may
> > be a good reason, and adding it in might cause problems.
>
> I think the period is stripped because recarrays also offer attribute
> access of names. So you wouldn't be able to do
>
> your_array.sample.txt
>
> All the names get passed through a name validator. IIRC it's something like
>
> from numpy.lib import _iotools
>
> validator = _iotools.NameValidator()
>
> validator.validate('sample1.txt')
> validator.validate('a name with spaces')
>
> NameValidator has a good docstring and the gist of this should be in
> the genfromtxt docs, if it's not already.
>
> Skipper
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Forbidden charcter in the "names" argument of genfromtxt?

[Skipper Seabold]

On Mon, Feb 20, 2012 at 1:35 PM, Brett Olsen  wrote:
> On Sat, Feb 18, 2012 at 8:12 PM, Adam Hughes  wrote:
>> Hey everyone,
>>
>> I have timeseries data in which the column label is simply a filename from
>> which the original data was taken.  Here's some sample data:
>>
>> name1.txt  name2.txt  name3.txt
>> 32  34    953
>> 32  03    402
>>
>> I've noticed that the standard genfromtxt() method works great; however, the
>> names aren't written correctly.  That is, if I use the command:
>>
>> print data['name1.txt']
>>
>> Nothing happens.
>>
>> However, when I remove the file extension, Eg:
>>
>> name1  name2  name3
>> 32  34    953
>> 32  03    402
>>
>> Then print data['name1'] return (32, 32) as expected.  It seems that the
>> period in the name isn't compatible with the genfromtxt() names attribute.
>> Is there a workaround, or do I need to restructure my program to get the
>> extension removed?  I'd rather not do this if possible for reasons that
>> aren't important for the discussion at hand.
>
> It looks like the period is just getting stripped out of the names:
>
> In [1]: import numpy as N
>
> In [2]: N.genfromtxt('sample.txt', names=True)
> Out[2]:
> array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
>      dtype=[('name1txt', '
> Interestingly, this still happens if you supply the names manually:
>
> In [17]: def reader(filename):
>   :     infile = open(filename, 'r')
>   :     names = infile.readline().split()
>   :     data = N.genfromtxt(infile, names=names)
>   :     infile.close()
>   :     return data
>   :
>
> In [20]: data = reader('sample.txt')
>
> In [21]: data
> Out[21]:
> array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
>      dtype=[('name1txt', '
> What you can do is reset the names after genfromtxt is through with it, 
> though:
>
> In [34]: def reader(filename):
>   :     infile = open(filename, 'r')
>   :     names = infile.readline().split()
>   :     infile.close()
>   :     data = N.genfromtxt(filename, names=True)
>   :     data.dtype.names = names
>   :     return data
>   :
>
> In [35]: data = reader('sample.txt')
>
> In [36]: data
> Out[36]:
> array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
>      dtype=[('name1.txt', '
> Be warned, I don't know why the period is getting stripped; there may
> be a good reason, and adding it in might cause problems.

I think the period is stripped because recarrays also offer attribute
access of names. So you wouldn't be able to do

your_array.sample.txt

All the names get passed through a name validator. IIRC it's something like

from numpy.lib import _iotools

validator = _iotools.NameValidator()

validator.validate('sample1.txt')
validator.validate('a name with spaces')

NameValidator has a good docstring and the gist of this should be in
the genfromtxt docs, if it's not already.

Skipper
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Forbidden charcter in the "names" argument of genfromtxt?

[Adam Hughes]

Thanks Brett.  I appreciate you taking the time to help me out.  In
particular, I did not know the correct syntax for this:

data.dtype.names = names

Which is very helpful.  If I would have known how to access
data.dtype.names, I think it would have saved me a great deal of trouble.
I guess it's all part of a learning curve.  I'll keep in mind that the
period may cause problems later; however, as far as I can tell so far,
there's nothing going wrong when I access the data.

On Mon, Feb 20, 2012 at 1:35 PM, Brett Olsen  wrote:

> On Sat, Feb 18, 2012 at 8:12 PM, Adam Hughes 
> wrote:
> > Hey everyone,
> >
> > I have timeseries data in which the column label is simply a filename
> from
> > which the original data was taken.  Here's some sample data:
> >
> > name1.txt  name2.txt  name3.txt
> > 32  34953
> > 32  03402
> >
> > I've noticed that the standard genfromtxt() method works great; however,
> the
> > names aren't written correctly.  That is, if I use the command:
> >
> > print data['name1.txt']
> >
> > Nothing happens.
> >
> > However, when I remove the file extension, Eg:
> >
> > name1  name2  name3
> > 32  34953
> > 32  03402
> >
> > Then print data['name1'] return (32, 32) as expected.  It seems that the
> > period in the name isn't compatible with the genfromtxt() names
> attribute.
> > Is there a workaround, or do I need to restructure my program to get the
> > extension removed?  I'd rather not do this if possible for reasons that
> > aren't important for the discussion at hand.
>
> It looks like the period is just getting stripped out of the names:
>
> In [1]: import numpy as N
>
> In [2]: N.genfromtxt('sample.txt', names=True)
> Out[2]:
> array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
>  dtype=[('name1txt', '
> Interestingly, this still happens if you supply the names manually:
>
> In [17]: def reader(filename):
>   : infile = open(filename, 'r')
>   : names = infile.readline().split()
>   : data = N.genfromtxt(infile, names=names)
>   : infile.close()
>   : return data
>   :
>
> In [20]: data = reader('sample.txt')
>
> In [21]: data
> Out[21]:
> array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
>  dtype=[('name1txt', '
> What you can do is reset the names after genfromtxt is through with it,
> though:
>
> In [34]: def reader(filename):
>   : infile = open(filename, 'r')
>   : names = infile.readline().split()
>   : infile.close()
>   : data = N.genfromtxt(filename, names=True)
>   : data.dtype.names = names
>   : return data
>   :
>
> In [35]: data = reader('sample.txt')
>
> In [36]: data
> Out[36]:
> array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
>  dtype=[('name1.txt', ' '
> Be warned, I don't know why the period is getting stripped; there may
> be a good reason, and adding it in might cause problems.
>
> ~Brett
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Forbidden charcter in the "names" argument of genfromtxt?

[Brett Olsen]

On Sat, Feb 18, 2012 at 8:12 PM, Adam Hughes  wrote:
> Hey everyone,
>
> I have timeseries data in which the column label is simply a filename from
> which the original data was taken.  Here's some sample data:
>
> name1.txt  name2.txt  name3.txt
> 32  34    953
> 32  03    402
>
> I've noticed that the standard genfromtxt() method works great; however, the
> names aren't written correctly.  That is, if I use the command:
>
> print data['name1.txt']
>
> Nothing happens.
>
> However, when I remove the file extension, Eg:
>
> name1  name2  name3
> 32  34    953
> 32  03    402
>
> Then print data['name1'] return (32, 32) as expected.  It seems that the
> period in the name isn't compatible with the genfromtxt() names attribute.
> Is there a workaround, or do I need to restructure my program to get the
> extension removed?  I'd rather not do this if possible for reasons that
> aren't important for the discussion at hand.

It looks like the period is just getting stripped out of the names:

In [1]: import numpy as N

In [2]: N.genfromtxt('sample.txt', names=True)
Out[2]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
  dtype=[('name1txt', 'http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Francesc Alted]

On Feb 20, 2012, at 7:08 PM, Dag Sverre Seljebotn wrote:

> On 02/20/2012 09:34 AM, Christopher Jordan-Squire wrote:
>> On Mon, Feb 20, 2012 at 9:18 AM, Dag Sverre Seljebotn
>>   wrote:
>>> On 02/20/2012 08:55 AM, Sturla Molden wrote:
 Den 20.02.2012 17:42, skrev Sturla Molden:
> There are still other options than C or C++ that are worth considering.
> One would be to write NumPy in Python. E.g. we could use LLVM as a
> JIT-compiler and produce the performance critical code we need on the fly.
> 
> 
 
 LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster
 than GCC and often produces better machine code. They can therefore be
 used inside an array library. It would give a faster NumPy, and we could
 keep most of it in Python.
>>> 
>>> I think it is moot to focus on improving NumPy performance as long as in
>>> practice all NumPy operations are memory bound due to the need to take a
>>> trip through system memory for almost any operation. C/C++ is simply
>>> "good enough". JIT is when you're chasing a 2x improvement or so, but
>>> today NumPy can be 10-20x slower than a Cython loop.
>>> 
>> 
>> I don't follow this. Could you expand a bit more? (Specifically, I
>> wasn't aware that numpy could be 10-20x slower than a cython loop, if
>> we're talking about the base numpy library--so core operations. I'm
> 
> The problem with NumPy is the temporaries needed -- if you want to compute
> 
> A + B + np.sqrt(D)
> 
> then, if the arrays are larger than cache size (a couple of megabytes), 
> then each of those operations will first transfer the data in and out 
> over the memory bus. I.e. first you compute an element of sqrt(D), then 
> the result of that is put in system memory, then later the same number 
> is read back in order to add it to an element in B, and so on.
> 
> The compute-to-bandwidth ratio of modern CPUs is between 30:1 and 
> 60:1... so in extreme cases it's cheaper to do 60 additions than to 
> transfer a single number from system memory.
> 
> It is much faster to only transfer an element (or small block) from each 
> of A, B, and D to CPU cache, then do the entire expression, then 
> transfer the result back. This is easy to code in Cython/Fortran/C and 
> impossible with NumPy/Python.
> 
> This is why numexpr/Theano exists.

Well, I can't speak for Theano (it is quite more general than numexpr, and more 
geared towards using GPUs, right?), but this was certainly the issue that make 
David Cooke to create numexpr.  A more in-deep explanation about this problem 
can be seen in:

http://www.euroscipy.org/talk/1657

which includes some graphical explanations.

-- Francesc Alted



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray and lazy evaluation

[Dag Sverre Seljebotn]

On 02/20/2012 10:04 AM, Francesc Alted wrote:
> On Feb 20, 2012, at 6:46 PM, Dag Sverre Seljebotn wrote:
>
>> On 02/20/2012 09:24 AM, Olivier Delalleau wrote:
>>> Hi Dag,
>>>
>>> Would you mind elaborating a bit on that example you mentioned at the
>>> end of your email? I don't quite understand what behavior you would like
>>> to achieve
>>
>> Sure, see below. I think we should continue discussion on numpy-discuss.
>>
>> I wrote:
>>
>>> You need at least a slightly different Python API to get anywhere, so
>>> numexpr/Theano is the right place to work on an implementation of this
>>> idea. Of course it would be nice if numexpr/Theano offered something as
>>> convenient as
>>>
>>> with lazy:
>>>  arr = A + B + C # with all of these NumPy arrays
>>> # compute upon exiting...
>>
>> More information:
>>
>> The disadvantage today of using Theano (or numexpr) is that they require
>> using a different API, so that one has to learn and use Theano "from the
>> ground up", rather than just slap it on in an optimization phase.
>>
>> The alternative would require extensive changes to NumPy, so I guess
>> Theano authors or Francesc would need to push for this.
>>
>> The alternative would be (with A, B, C ndarray instances):
>>
>> with theano.lazy:
>>  arr = A + B + C
>>
>> On __enter__, the context manager would hook into NumPy to override it's
>> arithmetic operators. Then it would build a Theano symbolic tree instead
>> of performing computations right away.
>>
>> In addition to providing support for overriding arithmetic operators,
>> slicing etc., it would be necesarry for "arr" to be an ndarray instance
>> which is "not yet computed" (data-pointer set to NULL, and store a
>> compute-me callback and some context information).
>>
>> Finally, the __exit__ would trigger computation. For other operations
>> which need the data pointer (e.g., single element lookup) one could
>> either raise an exception or trigger computation.
>>
>> This is just a rough sketch. It is not difficult "in principle", but of
>> course there's really a massive amount of work involved to work support
>> for this into the NumPy APIs.
>>
>> Probably, we're talking a NumPy 3.0 thing, after the current round of
>> refactorings have settled...
>>
>> Please: Before discussing this further one should figure out if there's
>> manpower available for it; no sense in hashing out a castle in the sky
>> in details.
>
> I see.  Mark Wiebe already suggested the same thing some time ago:
>
> https://github.com/numpy/numpy/blob/master/doc/neps/deferred-ufunc-evaluation.rst

Thanks, I didn't know about that (though I did really assume this was on 
Mark's radar already).

>
>> Also it would be better to talk in person about this if
>> possible (I'm in Berkeley now and will attend PyData and PyCon).
>
> Nice.  Most of Continuum crew (me included) will be attending to both 
> conferences.  Mark W. will make PyCon only, but will be a good occasion to 
> discuss this further.

I certainly don't think I have anything to add to this discussion beyond 
what Mark wrote up. But will be nice to meet up anyway.

Dag
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Dag Sverre Seljebotn]

On 02/20/2012 09:34 AM, Christopher Jordan-Squire wrote:
> On Mon, Feb 20, 2012 at 9:18 AM, Dag Sverre Seljebotn
>   wrote:
>> On 02/20/2012 08:55 AM, Sturla Molden wrote:
>>> Den 20.02.2012 17:42, skrev Sturla Molden:
 There are still other options than C or C++ that are worth considering.
 One would be to write NumPy in Python. E.g. we could use LLVM as a
 JIT-compiler and produce the performance critical code we need on the fly.


>>>
>>> LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster
>>> than GCC and often produces better machine code. They can therefore be
>>> used inside an array library. It would give a faster NumPy, and we could
>>> keep most of it in Python.
>>
>> I think it is moot to focus on improving NumPy performance as long as in
>> practice all NumPy operations are memory bound due to the need to take a
>> trip through system memory for almost any operation. C/C++ is simply
>> "good enough". JIT is when you're chasing a 2x improvement or so, but
>> today NumPy can be 10-20x slower than a Cython loop.
>>
>
> I don't follow this. Could you expand a bit more? (Specifically, I
> wasn't aware that numpy could be 10-20x slower than a cython loop, if
> we're talking about the base numpy library--so core operations. I'm

The problem with NumPy is the temporaries needed -- if you want to compute

A + B + np.sqrt(D)

then, if the arrays are larger than cache size (a couple of megabytes), 
then each of those operations will first transfer the data in and out 
over the memory bus. I.e. first you compute an element of sqrt(D), then 
the result of that is put in system memory, then later the same number 
is read back in order to add it to an element in B, and so on.

The compute-to-bandwidth ratio of modern CPUs is between 30:1 and 
60:1... so in extreme cases it's cheaper to do 60 additions than to 
transfer a single number from system memory.

It is much faster to only transfer an element (or small block) from each 
of A, B, and D to CPU cache, then do the entire expression, then 
transfer the result back. This is easy to code in Cython/Fortran/C and 
impossible with NumPy/Python.

This is why numexpr/Theano exists.

You can make the slowdown over Cython/Fortran/C almost arbitrarily large 
by adding terms to the equation above. So of course, the actual slowdown 
depends on your usecase.

> also not totally sure why a JIT is a 2x improvement or so vs. cython.
> Not that a disagree on either of these points, I'd just like a bit
> more detail.)

I meant that the JIT may be a 2x improvement over the current NumPy C 
code. There's some logic when iterating arrays that could perhaps be 
specialized away depending on the actual array layout at runtime.

But I'm thinking that probably a JIT wouldn't help all that much, so 
it's probably 1x -- the 2x was just to be very conservative w.r.t. the 
argument I was making, as I don't know the NumPy C sources well enough.

Dag
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray and lazy evaluation (was: Proposed Rodmap Overview)

[Olivier Delalleau]

Never mind. The link Francesc posted answered my question :)

-=- Olivier

Le 20 février 2012 12:54, Olivier Delalleau  a
écrit :

> Le 20 février 2012 12:46, Dag Sverre Seljebotn  > a écrit :
>
> On 02/20/2012 09:24 AM, Olivier Delalleau wrote:
>> > Hi Dag,
>> >
>> > Would you mind elaborating a bit on that example you mentioned at the
>> > end of your email? I don't quite understand what behavior you would like
>> > to achieve
>>
>> Sure, see below. I think we should continue discussion on numpy-discuss.
>>
>> I wrote:
>>
>> > You need at least a slightly different Python API to get anywhere, so
>> > numexpr/Theano is the right place to work on an implementation of this
>> > idea. Of course it would be nice if numexpr/Theano offered something as
>> > convenient as
>> >
>> > with lazy:
>> >  arr = A + B + C # with all of these NumPy arrays
>> > # compute upon exiting...
>>
>> More information:
>>
>> The disadvantage today of using Theano (or numexpr) is that they require
>> using a different API, so that one has to learn and use Theano "from the
>> ground up", rather than just slap it on in an optimization phase.
>>
>> The alternative would require extensive changes to NumPy, so I guess
>> Theano authors or Francesc would need to push for this.
>>
>> The alternative would be (with A, B, C ndarray instances):
>>
>> with theano.lazy:
>> arr = A + B + C
>>
>> On __enter__, the context manager would hook into NumPy to override it's
>> arithmetic operators. Then it would build a Theano symbolic tree instead
>> of performing computations right away.
>>
>> In addition to providing support for overriding arithmetic operators,
>> slicing etc., it would be necesarry for "arr" to be an ndarray instance
>> which is "not yet computed" (data-pointer set to NULL, and store a
>> compute-me callback and some context information).
>>
>> Finally, the __exit__ would trigger computation. For other operations
>> which need the data pointer (e.g., single element lookup) one could
>> either raise an exception or trigger computation.
>>
>> This is just a rough sketch. It is not difficult "in principle", but of
>> course there's really a massive amount of work involved to work support
>> for this into the NumPy APIs.
>>
>> Probably, we're talking a NumPy 3.0 thing, after the current round of
>> refactorings have settled...
>>
>> Please: Before discussing this further one should figure out if there's
>> manpower available for it; no sense in hashing out a castle in the sky
>> in details. Also it would be better to talk in person about this if
>> possible (I'm in Berkeley now and will attend PyData and PyCon).
>>
>> Dag
>>
>
> Thanks for the additional details.
>
> I feel like this must be a stupid question, but I have to ask: what is the
> point of being lazy here, since the computation is performed on exit anyway?
>
> -=- Olivier
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ndarray and lazy evaluation (was: Proposed Rodmap Overview)

[Francesc Alted]

On Feb 20, 2012, at 6:46 PM, Dag Sverre Seljebotn wrote:

> On 02/20/2012 09:24 AM, Olivier Delalleau wrote:
>> Hi Dag,
>> 
>> Would you mind elaborating a bit on that example you mentioned at the
>> end of your email? I don't quite understand what behavior you would like
>> to achieve
> 
> Sure, see below. I think we should continue discussion on numpy-discuss.
> 
> I wrote:
> 
>> You need at least a slightly different Python API to get anywhere, so
>> numexpr/Theano is the right place to work on an implementation of this
>> idea. Of course it would be nice if numexpr/Theano offered something as
>> convenient as
>> 
>> with lazy:
>> arr = A + B + C # with all of these NumPy arrays
>> # compute upon exiting...
> 
> More information:
> 
> The disadvantage today of using Theano (or numexpr) is that they require 
> using a different API, so that one has to learn and use Theano "from the 
> ground up", rather than just slap it on in an optimization phase.
> 
> The alternative would require extensive changes to NumPy, so I guess 
> Theano authors or Francesc would need to push for this.
> 
> The alternative would be (with A, B, C ndarray instances):
> 
> with theano.lazy:
> arr = A + B + C
> 
> On __enter__, the context manager would hook into NumPy to override it's 
> arithmetic operators. Then it would build a Theano symbolic tree instead 
> of performing computations right away.
> 
> In addition to providing support for overriding arithmetic operators, 
> slicing etc., it would be necesarry for "arr" to be an ndarray instance 
> which is "not yet computed" (data-pointer set to NULL, and store a 
> compute-me callback and some context information).
> 
> Finally, the __exit__ would trigger computation. For other operations 
> which need the data pointer (e.g., single element lookup) one could 
> either raise an exception or trigger computation.
> 
> This is just a rough sketch. It is not difficult "in principle", but of 
> course there's really a massive amount of work involved to work support 
> for this into the NumPy APIs.
> 
> Probably, we're talking a NumPy 3.0 thing, after the current round of 
> refactorings have settled...
> 
> Please: Before discussing this further one should figure out if there's 
> manpower available for it; no sense in hashing out a castle in the sky 
> in details.

I see.  Mark Wiebe already suggested the same thing some time ago:

https://github.com/numpy/numpy/blob/master/doc/neps/deferred-ufunc-evaluation.rst

> Also it would be better to talk in person about this if 
> possible (I'm in Berkeley now and will attend PyData and PyCon).

Nice.  Most of Continuum crew (me included) will be attending to both 
conferences.  Mark W. will make PyCon only, but will be a good occasion to 
discuss this further.

See you,

-- Francesc Alted



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Sturla Molden]

Den 20.02.2012 18:34, skrev Christopher Jordan-Squire:
> I don't follow this. Could you expand a bit more? (Specifically, I 
> wasn't aware that numpy could be 10-20x slower than a cython loop, if 
> we're talking about the base numpy library--so core operations. I'm 
> also not totally sure why a JIT is a 2x improvement or so vs. cython. 
> Not that a disagree on either of these points, I'd just like a bit 
> more detail.) 

Dag Sverre is right about this.

NumPy is memory bound, Cython loops are (usually) CPU bound.

If you write:

 x[:] = a + b + c  # numpy arrays

then this happens (excluding reference counting):

- allocate temporary array
- loop over a and b, add to temporary
- allocate 2nd temporary array
- loop over 1st temporary array  and c, add to 2nd
- deallocate 1st temporary array
- loop over 2nd temporary array, assign to x
- deallocate 2nd temporary array

Since memory access is slow, memory allocation and deallocation
is slow, and computation is fast, this will be perhaps 10 times
slower than what we could do with a loop in Cython:

 for i in range(n):
 x[i] = a[i] + b[i] + c[i]

I.e. we get rid of the temporary arrays and the multiple loops.
All the temporaries here are put in registers.

It is streaming data into the CPU that is slow, not computing!

It has actually been experimented with streaming data in a
compressed form, and decompressing on the fly, as data access
still dominates the runtime (even if you do a lot of computing
per element).


Sturla
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ndarray and lazy evaluation (was: Proposed Rodmap Overview)

[Dag Sverre Seljebotn]

On 02/20/2012 09:24 AM, Olivier Delalleau wrote:
> Hi Dag,
>
> Would you mind elaborating a bit on that example you mentioned at the
> end of your email? I don't quite understand what behavior you would like
> to achieve

Sure, see below. I think we should continue discussion on numpy-discuss.

I wrote:

> You need at least a slightly different Python API to get anywhere, so
> numexpr/Theano is the right place to work on an implementation of this
> idea. Of course it would be nice if numexpr/Theano offered something as
> convenient as
>
> with lazy:
>  arr = A + B + C # with all of these NumPy arrays
> # compute upon exiting...

More information:

The disadvantage today of using Theano (or numexpr) is that they require 
using a different API, so that one has to learn and use Theano "from the 
ground up", rather than just slap it on in an optimization phase.

The alternative would require extensive changes to NumPy, so I guess 
Theano authors or Francesc would need to push for this.

The alternative would be (with A, B, C ndarray instances):

with theano.lazy:
 arr = A + B + C

On __enter__, the context manager would hook into NumPy to override it's 
arithmetic operators. Then it would build a Theano symbolic tree instead 
of performing computations right away.

In addition to providing support for overriding arithmetic operators, 
slicing etc., it would be necesarry for "arr" to be an ndarray instance 
which is "not yet computed" (data-pointer set to NULL, and store a 
compute-me callback and some context information).

Finally, the __exit__ would trigger computation. For other operations 
which need the data pointer (e.g., single element lookup) one could 
either raise an exception or trigger computation.

This is just a rough sketch. It is not difficult "in principle", but of 
course there's really a massive amount of work involved to work support 
for this into the NumPy APIs.

Probably, we're talking a NumPy 3.0 thing, after the current round of 
refactorings have settled...

Please: Before discussing this further one should figure out if there's 
manpower available for it; no sense in hashing out a castle in the sky 
in details. Also it would be better to talk in person about this if 
possible (I'm in Berkeley now and will attend PyData and PyCon).

Dag
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Sturla Molden]

Den 20.02.2012 18:18, skrev Dag Sverre Seljebotn:
>
> I think it is moot to focus on improving NumPy performance as long as in
> practice all NumPy operations are memory bound due to the need to take a
> trip through system memory for almost any operation. C/C++ is simply
> "good enough". JIT is when you're chasing a 2x improvement or so, but
> today NumPy can be 10-20x slower than a Cython loop.
>
> You need at least a slightly different Python API to get anywhere, so
> numexpr/Theano is the right place to work on an implementation of this
> idea. Of course it would be nice if numexpr/Theano offered something as
> convenient as
>
> with lazy:
>   arr = A + B + C # with all of these NumPy arrays
> # compute upon exiting...
>
>

Lazy evaluation is nice. But I was thinking more about how to avoid C++ 
in the NumPy core, so more than 2 or 3 programmers could contribute.

I.e. my point was not that loops in LLVM would be much faster than C++ 
(that is besides the point), but the code could be written in Python 
instead of C++.

But if the idea is to support other languages as well (which I somehow 
forgot), then this approach certainly becomes less useful.

(OTOH, lazy evaluation is certainly easier to achieve with JIT 
compilation. But that will have to wait until NumPy 5.0 perhaps...)


Sturla









___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Christopher Jordan-Squire]

On Mon, Feb 20, 2012 at 9:18 AM, Dag Sverre Seljebotn
 wrote:
> On 02/20/2012 08:55 AM, Sturla Molden wrote:
>> Den 20.02.2012 17:42, skrev Sturla Molden:
>>> There are still other options than C or C++ that are worth considering.
>>> One would be to write NumPy in Python. E.g. we could use LLVM as a
>>> JIT-compiler and produce the performance critical code we need on the fly.
>>>
>>>
>>
>> LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster
>> than GCC and often produces better machine code. They can therefore be
>> used inside an array library. It would give a faster NumPy, and we could
>> keep most of it in Python.
>
> I think it is moot to focus on improving NumPy performance as long as in
> practice all NumPy operations are memory bound due to the need to take a
> trip through system memory for almost any operation. C/C++ is simply
> "good enough". JIT is when you're chasing a 2x improvement or so, but
> today NumPy can be 10-20x slower than a Cython loop.
>

I don't follow this. Could you expand a bit more? (Specifically, I
wasn't aware that numpy could be 10-20x slower than a cython loop, if
we're talking about the base numpy library--so core operations. I'm
also not totally sure why a JIT is a 2x improvement or so vs. cython.
Not that a disagree on either of these points, I'd just like a bit
more detail.)

Thanks,
Chris

> You need at least a slightly different Python API to get anywhere, so
> numexpr/Theano is the right place to work on an implementation of this
> idea. Of course it would be nice if numexpr/Theano offered something as
> convenient as
>
> with lazy:
>     arr = A + B + C # with all of these NumPy arrays
> # compute upon exiting...
>
> Dag
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Matthieu Brucher]

2012/2/19 Sturla Molden 

> Den 19.02.2012 10:28, skrev Mark Wiebe:
> >
> > Particular styles of using templates can cause this, yes. To properly
> > do this kind of advanced C++ library work, it's important to think
> > about the big-O notation behavior of your template instantiations, not
> > just the big-O notation of run-time. C++ templates have a
> > turing-complete language (which is said to be quite similar to
> > haskell, but spelled vastly different) running at compile time in
> > them. This is what gives template meta-programming in C++ great power,
> > but since templates weren't designed for this style of programming
> > originally, template meta-programming is not very easy.
> >
> >
>
> The problem with metaprogramming is that we are doing manually the work
> that belongs to the compiler. Blitz++ was supposed to be a library that
> "thought like a compiler". But then compilers just got better. Today, it
> is no longer possible for a numerical library programmer to outsmart an
> optimizing C++ compiler. All metaprogramming can do today is produce
> error messages noone can understand. And the resulting code will often
> be slower because the compiler has less opportunities to do its work.
>

As I've said, the compiler is pretty much stupid. It cannot do what
Blitzz++ did, or what Eigen is currently doing, mainly because of the basis
different languages (C or C++).

-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Francesc Alted]

On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote:
> You need at least a slightly different Python API to get anywhere, so 
> numexpr/Theano is the right place to work on an implementation of this 
> idea. Of course it would be nice if numexpr/Theano offered something as 
> convenient as
> 
> with lazy:
> arr = A + B + C # with all of these NumPy arrays
> # compute upon exiting…

Hmm, that would be cute indeed.  Do you have an idea on how the code in the 
with context could be passed to the Python AST compiler (à la 
numexpr.evaluate("A + B + C"))?

-- Francesc Alted



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Matthieu Brucher]

2012/2/19 Nathaniel Smith 

> On Sun, Feb 19, 2012 at 9:16 AM, David Cournapeau 
> wrote:
> > On Sun, Feb 19, 2012 at 8:08 AM, Mark Wiebe  wrote:
> >> Is there a specific
> >> target platform/compiler combination you're thinking of where we can do
> >> tests on this? I don't believe the compile times are as bad as many
> people
> >> suspect, can you give some simple examples of things we might do in
> NumPy
> >> you expect to compile slower in C++ vs C?
> >
> > Switching from gcc to g++ on the same codebase should not change much
> > compilation times. We should test, but that's not what worries me.
> > What worries me is when we start using C++ specific code, STL and co.
> > Today, scipy.sparse.sparsetools takes half of the build time  of the
> > whole scipy, and it does not even use fancy features. It also takes Gb
> > of ram when building in parallel.
>
> I like C++ but it definitely does have issues with compilation times.
>
> IIRC the main problem is very simple: STL and friends (e.g. Boost) are
> huge libraries, and because they use templates, the entire source code
> is in the header files. That means that as soon as you #include a few
> standard C++ headers, your innocent little source file has suddenly
> become hundreds of thousands of lines long, and it just takes the
> compiler a while to churn through megabytes of source code, no matter
> what it is. (Effectively you recompile some significant fraction of
> STL from scratch on every file, and then throw it away.)
>

In fact Boost tries to be clean about this. Up to a few minor releases of
GCC, their headers were a mess. When you included something, a lot of
additional code was brought, and the compile-time exploded. But this is no
longer the case. If we restrict the core to a few includes, even with
templates, it should not be long to compile.

-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Sturla Molden]

Den 20.02.2012 18:14, skrev Charles R Harris:
>
> Would that work for Ruby also? One of the advantages of C++ is that 
> the code doesn't need to be refactored to start with, just modified 
> step by step going into the future. I think PyPy is close to what you 
> are talking about.
>

If we plant to support more languages than Python, it might be better to 
use C++ (sorry).

But it does not mean that LLVM cannot be used. Either one can generate C 
or C++, or just use the assembly language (which is very simple and 
readable too: http://llvm.org/docs/LangRef.html).

We have exact knowledge about an ndarray at runtime:

- dtype
- dimensions
- strides
- whether the array is contiguous or not

This can be JIT-compiled into specialized looping code by LLVM. These 
kernels can then be stored in a database and resued.

If it matters, LLVM is embeddable in C++.


Sturla








___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Matthieu Brucher]

2012/2/19 Matthew Brett 

> Hi,
>
> On Sat, Feb 18, 2012 at 8:38 PM, Travis Oliphant 
> wrote:
>
> > We will need to see examples of what Mark is talking about and clarify
> some
> > of the compiler issues.   Certainly there is some risk that once code is
> > written that it will be tempting to just use it.   Other approaches are
> > certainly worth exploring in the mean-time, but C++ has some strong
> > arguments for it.
>
> The worry as I understand it is that a C++ rewrite might make the
> numpy core effectively a read-only project for anyone but Mark.  Do
> you have any feeling for whether that is likely?
>

Some of us are C developers, other are C++. It will depend on the
background of each of us.


> How would numpylib compare to libraries like eigen?  How likely do you
> think it would be that unrelated projects would use numpylib rather
> than eigen or other numerical libraries?  Do you think the choice of
> C++ rather than C will influence whether other projects will take it
> up?
>

I guess that the C++ port may open a door to change the back-end, and
perhaps use Eigen, or ArBB. As those guys (ArBB) wanted to provided a
Python interface compatible with Numpy to their VM, it may be interesting
to be able to change back-ends (although it is limited to one platform and
2 OS).

-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Matthieu Brucher]

> Would it be fair to say then, that you are expecting the discussion
> about C++ will mainly arise after the Mark has written the code?   I
> can see that it will be easier to specific at that point, but there
> must be a serious risk that it will be too late to seriously consider
> an alternative approach.
>
>
> We will need to see examples of what Mark is talking about and clarify
> some of the compiler issues.   Certainly there is some risk that once code
> is written that it will be tempting to just use it.   Other approaches are
> certainly worth exploring in the mean-time, but C++ has some strong
> arguments for it.
>

Compilers for C++98 are now stable enough (except on Bluegene, see the
Boost distribution with xlc++)
C++ helps a lot to enhance robustness.ts?

>
> From my perspective having a standalone core NumPy is still a goal.   The
> primary advantages of having a NumPy library (call it NumLib for the sake
> of argument) are
>
> 1) Ability for projects like PyPy, IronPython, and Jython to use it more
> easily
> 2) Ability for Ruby, Perl, Node.JS, and other new languages to use the
> code for their technical computing projects.
> 3) increasing the number of users who can help make it more solid
> 4) being able to build the user-base (and corresponding performance with
> eye-balls from Intel, NVidia, AMD, Microsoft, Google, etc. looking at the
> code).
>
> The disadvantages I can think of:
>  1) More users also means we might risk "lowest-commond-denominator"
> problems --- i.e. trying to be too much to too many may make it not useful
> for anyone. Also, more users means more people with opinions that might be
> difficult to re-concile.
> 2) The work of doing the re-write is not small:  probably at least 6
> person-months
> 3) Not being able to rely on Python objects (dictionaries, lists, and
> tuples are currently used in the code-base quite a bit --- though the
> re-factor did show some examples of how to remove this usage).
> 4) Handling of "Object" arrays requires some re-design.
>
> I'm sure there are other factors that could be added to both lists.
>
> -Travis
>
>
>
> Thanks a lot for the reply,
>
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Matthieu Brucher]

> C++11 has this option:
>
> for (auto& item : container) {
> // iterate over the container object,
> // get a reference to each item
> //
> // "container" can be an STL class or
> // A C-style array with known size.
> }
>
> Which does this:
>
> for item in container:
> pass
>

It is even better than using the macro way because the compiler knows
everything is constant (start and end), so it can do better things.


> > Using C++ templates to generate ufunc loops is an obvious application,
> > but again, in the simple examples
>
> Template metaprogramming?
>
> Don't even think about it. It is brain dead to try to outsmart the
> compiler.
>

It is really easy to outsmart the compiler. Really. I use metaprogramming
for loop creation to optimize cache behavior, communication in parallel
environments, and there is no way the compiler would have done things as
efficiently (and there is a lot of leeway to enhance my code).

-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Dag Sverre Seljebotn]

On 02/20/2012 08:55 AM, Sturla Molden wrote:
> Den 20.02.2012 17:42, skrev Sturla Molden:
>> There are still other options than C or C++ that are worth considering.
>> One would be to write NumPy in Python. E.g. we could use LLVM as a
>> JIT-compiler and produce the performance critical code we need on the fly.
>>
>>
>
> LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster
> than GCC and often produces better machine code. They can therefore be
> used inside an array library. It would give a faster NumPy, and we could
> keep most of it in Python.

I think it is moot to focus on improving NumPy performance as long as in 
practice all NumPy operations are memory bound due to the need to take a 
trip through system memory for almost any operation. C/C++ is simply 
"good enough". JIT is when you're chasing a 2x improvement or so, but 
today NumPy can be 10-20x slower than a Cython loop.

You need at least a slightly different Python API to get anywhere, so 
numexpr/Theano is the right place to work on an implementation of this 
idea. Of course it would be nice if numexpr/Theano offered something as 
convenient as

with lazy:
 arr = A + B + C # with all of these NumPy arrays
# compute upon exiting...

Dag
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Charles R Harris]

On Mon, Feb 20, 2012 at 9:55 AM, Sturla Molden  wrote:

> Den 20.02.2012 17:42, skrev Sturla Molden:
> > There are still other options than C or C++ that are worth considering.
> > One would be to write NumPy in Python. E.g. we could use LLVM as a
> > JIT-compiler and produce the performance critical code we need on the
> fly.
> >
> >
>
> LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster
> than GCC and often produces better machine code. They can therefore be
> used inside an array library. It would give a faster NumPy, and we could
> keep most of it in Python.
>
>
Would that work for Ruby also? One of the advantages of C++ is that the
code doesn't need to be refactored to start with, just modified step by
step going into the future. I think PyPy is close to what you are talking
about.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Sturla Molden]

Den 20.02.2012 17:42, skrev Sturla Molden:
> There are still other options than C or C++ that are worth considering.
> One would be to write NumPy in Python. E.g. we could use LLVM as a
> JIT-compiler and produce the performance critical code we need on the fly.
>
>

LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster 
than GCC and often produces better machine code. They can therefore be 
used inside an array library. It would give a faster NumPy, and we could 
keep most of it in Python.

Sturla

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Sturla Molden]

Den 19.02.2012 00:09, skrev David Cournapeau:
> There are better languages than C++ that has most of the technical 
> benefits stated in this discussion (rust and D being the most 
> "obvious" ones), but whose usage is unrealistic today for various 
> reasons: knowledge, availability on "esoteric" platforms, etc… A new 
> language is completely ridiculous.

There are still other options than C or C++ that are worth considering. 
One would be to write NumPy in Python. E.g. we could use LLVM as a 
JIT-compiler and produce the performance critical code we need on the fly.

Sturla



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Sturla Molden]

Den 20.02.2012 08:35, skrev Paul Anton Letnes:
> As far as I can understand, implementing element-wise operations, slicing, 
> and a host of other NumPy features is in some sense pointless - the Fortran 
> compiler authors have already done it for us.

Only if you know the array dimensions in advance.

Sturla



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Sturla Molden]

Den 20.02.2012 08:35, skrev Paul Anton Letnes:
> In the language wars, I have one question. Why is Fortran not being 
> considered? Fortran already implements many of the features that we want in 
> NumPy:

Yes ... but it does not make Fortran a systems programming language. 
Making NumPy is different from using it.

> - slicing and similar operations, at least some of the fancy indexing kind
> - element-wise array operations and function calls
> - array bounds-checking and other debugging aid (with debugging flags)

That is nice for numerical computing, but not really needed to make NumPy.


> - arrays that mentally map very well onto numpy arrays. To me, this spells +1 
> to ease of contribution, over some abstract C/C++ template

Mentally perhaps, but not binary. NumPy needs uniformly strided memory 
on the binary level. Fortran just gives this at the mental level. E.g. 
there is nothing that dictates a Fortran pointer has to be a view, the 
compiler is free to employ copy-in copy-out. In Fortran, a function call 
can invalidate a pointer.  One would therefore have to store the array 
in an array of integer*1, and use the intrinsic function transfer() to 
parse the contents into NumPy dtypes.

> - in newer standards it has some nontrivial mathematical functions: gamma, 
> bessel, etc. that numpy lacks right now

That belongs to SciPy.


> - compilers that are good at optimizing for floating-point performance, 
> because that's what Fortran is all about

Insanely good, but not when we start to do the (binary, not mentally) 
strided access that NumPy needs. (Not that C compilers would be any better.)



> - not Fortran as such, but BLAS and LAPACK are easily accessed by Fortran
> - possibly other numerical libraries that can be helpful
> - Fortran has, in its newer standards, thought of C interoperability. We 
> could still keep bits of the code in C (or even C++?) if we'd like to, or 
> perhaps f2py/Cython could do the wrapping.

Not f2py, as it depends on NumPy.

- some programmers know Fortran better than C++. Fortran is at least used by 
many science guys, like me.


That is a valid arguments. Fortran is also much easier to read and debug.


Sturla





___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Sturla Molden]

Den 20.02.2012 10:54, skrev Pauli Virtanen:
> Fortran is OK for simple numerical algorithms, but starts to suck 
> heavily if you need to do any string handling, I/O, complicated logic, 
> or data structures

For string handling, C is actually worse than Fortran. In Fortran a 
string can be sliced like in Python. It is not as nice as Python, but 
far better than C.

Fortran's built-in I/O syntax is archaic, but the ISO C bindings in 
Fortran 2003 means one can use other means of I/O (posix, win api, C 
stdio) in a portable way.

Sturla


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Sturla Molden]

Den 20.02.2012 12:43, skrev Charles R Harris:
>
>
> There also used to be a problem with unsigned types not being 
> available. I don't know if that is still the case.
>

Fortran -- like Python and Java -- does not have built-in unsigned 
integer types. It is never really a problem though. One can e.g. use a 
longer integer or keep them in an array of bytes.

(Fortran 2003 is OOP so it is possible to define one if needed. Not 
saying it is a good idea.)


Sturla



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Charles R Harris]

On Mon, Feb 20, 2012 at 2:54 AM, Pauli Virtanen  wrote:

> 20.02.2012 08:35, Paul Anton Letnes kirjoitti:
> > In the language wars, I have one question.
> > Why is Fortran not being considered?
>
> Fortran is OK for simple numerical algorithms, but starts to suck
> heavily if you need to do any string handling, I/O, complicated logic,
> or data structures.
>
> Most of the work in Numpy implementation is not actually in numerics,
> but in figuring out the correct operation to dispatch the computations
> to. So, this is one reason why Fortran is not considered.
>
>
There also used to be a problem with unsigned types not being available. I
don't know if that is still the case.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Stéfan van der Walt]

On Mon, Feb 20, 2012 at 1:54 AM, Pauli Virtanen  wrote:
> 20.02.2012 08:35, Paul Anton Letnes kirjoitti:
>> In the language wars, I have one question.
>> Why is Fortran not being considered?
>
> Fortran is OK for simple numerical algorithms, but starts to suck
> heavily if you need to do any string handling, I/O, complicated logic,
> or data structures.

Out of curiosity, is this still true for the latest Fortran versions?
I guess there the problem may be compiler support over various
platforms.

Stéfan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Pauli Virtanen]

20.02.2012 08:35, Paul Anton Letnes kirjoitti:
> In the language wars, I have one question. 
> Why is Fortran not being considered?

Fortran is OK for simple numerical algorithms, but starts to suck
heavily if you need to do any string handling, I/O, complicated logic,
or data structures.

Most of the work in Numpy implementation is not actually in numerics,
but in figuring out the correct operation to dispatch the computations
to. So, this is one reason why Fortran is not considered.

-- 
Pauli Virtanen

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposed Roadmap Overview

[Samuel John]

On 17.02.2012, at 21:46, Ralf Gommers wrote:
> [...]
> So far no one has managed to build the numpy/scipy combo with the LLVM-based 
> compilers, so if you were willing to have a go at fixing that it would be 
> hugely appreciated. See http://projects.scipy.org/scipy/ticket/1500 for 
> details.
> 
> Once that's fixed, numpy can switch to using it for releases.

Well, I had great success with using clang and clang++ (which uses llvm) to 
compile both numpy and scipy on OS X 10.7.3.

Samuel

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion