Re: [Numpy-discussion] better error message possible?
On Mon, Jun 4, 2012 at 11:49 PM, Nathaniel Smith wrote: > On Mon, Jun 4, 2012 at 10:00 PM, Thouis (Ray) Jones wrote: >> On Mon, Jun 4, 2012 at 4:27 PM, Thouis (Ray) Jones wrote: >>> I could look into this. There are only ~10 places the code generates >>> this error, so it should be a pretty minor change. >> >> My initial estimate was low, but not overly so. An initial pass at >> adding index/dimension information to IndexErrors is here: >> https://github.com/thouis/numpy/tree/index_error_info > > Fabulous! I made a few comments there, but also: > >> A typical result: >> > numpy.zeros(3)[5] >> Traceback (most recent call last): >> File "", line 1, in >> IndexError: index 5 out of bounds in dimension 0 > > I would say "for", not "in". > > "index 5" is a bit ambiguous too... people might mis-read it as the > dimension, like, "the 5th index value I gave"? Not sure how to make it > unambiguous. Maybe: > > "IndexError: dimension 0 index out of bounds: got 5, size is 3" > > ? How about: IndexError: 5 is out of bounds for dimension 0: must be in [-3, 3). to be maximally explicit about what values are allowed, and avoid the "index" confusion. Ray Jones ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] nditer_buffer_flag branch (was: Add data memory allocation tracing facilities. (#284))
On Tue, Jun 5, 2012 at 11:06 AM, Thouis (Ray) Jones wrote: > All of the failing tests seem to have been caused by the buffer copy bug, > fixed in https://github.com/mwiebe/numpy/tree/nditer_buffer_flag (but not > yet pulled into numpy). > > I also have a version that implements tracing, with pure C in the allocation > functions writing to a dynamically allocated buffer, which must then be > fetched proactively by Python. However, I think this version is a little > nicer to use from the Python perspective. > > --- > Reply to this email directly or view it on GitHub: > https://github.com/numpy/numpy/pull/284#issuecomment-6121817 Speaking of which, Mark - what's the status of that nditer_buffer_flag branch? Should there be a pull request? -N ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] lazy evaluation
Hey, Another discussion on lazy evaluation, given the recent activity here: https://github.com/ContinuumIO/numba/pull/6#issuecomment-6117091 A somewhat recent previous thread can be found here: http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060862.html , and a NEP here: https://github.com/numpy/numpy/blob/master/doc/neps/deferred-ufunc-evaluation.rst I think trying to parse bytecode and build an expression graph for array expressions from that has disadvantages and is harder in general. For instance it won't be able to deal with branching at execution time, and things like inter-procedural analysis will be harder (not to mention you'd have to parse dtype creation). Instead, what you really want to do is hook into a lazy evaluating version of numpy, and generate your own code from the operations it records. It would be great if we implement the NEP listed above, but with a few extensions. I think Numpy should handle the lazy evaluation part, and determine when expressions should be evaluated, etc. However, for each user operation, Numpy will call back a user-installed hook implementing some interface, to allow various packages to provide their own hooks to evaluate vector operations however they want. This will include packages such as Theano, which could run things on the GPU, Numexpr, and in the future https://github.com/markflorisson88/minivect (which will likely have an LLVM backend in the future, and possibly integrated with Numba to allow inlining of numba ufuncs). The project above tries to bring together all the different array expression compilers together in a single framework, to provide efficient array expressions specialized for any data layout (nditer on steroids if you will, with SIMD, threaded and inlining capabilities). We could allow each hook to specify which dtypes it supports, and a minimal data size needed before it should be invoked (to avoid overhead for small arrays, like the openmp 'if' clause). If an operation is not supported, it will simply raise NotImplementedError, which means Numpy will evaluate the expression built so far and run its own implementation, resulting in a non-lazy array. E.g. if a library supports adding things together, but doesn't support the 'sin' function, np.sin(a + b) will result in the library executing a + b, and numpy evaluating sin on the result. So the idea is that the numpy lazy array will wrap an expression graph, which is built when the user performs operations and evaluated when needed (when a result is required or when someone tells numpy to evaluate all lazy arrays). Numpy will simply use the first hook willing to operate on data of the specified size and dtype, and will keep using that hook to build the expression until evaluated. Anyway, this is somewhat of a high-level overview. If there is any interest, we can flesh out the details and extend the NEP. Mark ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] better error message possible?
On Tue, Jun 5, 2012 at 12:15 PM, Thouis Jones wrote: > On Mon, Jun 4, 2012 at 11:49 PM, Nathaniel Smith wrote: >> On Mon, Jun 4, 2012 at 10:00 PM, Thouis (Ray) Jones wrote: >>> On Mon, Jun 4, 2012 at 4:27 PM, Thouis (Ray) Jones wrote: I could look into this. There are only ~10 places the code generates this error, so it should be a pretty minor change. >>> >>> My initial estimate was low, but not overly so. An initial pass at >>> adding index/dimension information to IndexErrors is here: >>> https://github.com/thouis/numpy/tree/index_error_info >> >> Fabulous! I made a few comments there, but also: >> >>> A typical result: >>> >> numpy.zeros(3)[5] >>> Traceback (most recent call last): >>> File "", line 1, in >>> IndexError: index 5 out of bounds in dimension 0 >> >> I would say "for", not "in". >> >> "index 5" is a bit ambiguous too... people might mis-read it as the >> dimension, like, "the 5th index value I gave"? Not sure how to make it >> unambiguous. Maybe: >> >> "IndexError: dimension 0 index out of bounds: got 5, size is 3" >> >> ? > > How about: > IndexError: 5 is out of bounds for dimension 0: must be in [-3, 3). > > to be maximally explicit about what values are allowed, and avoid the > "index" confusion. Or perhaps "axis" instead of "dimension", since this is how they are referred to in most numpy argument lists. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] varargs for logical_or, etc
I think it's unfortunate that functions like logical_or are limited to binary. As a workaround, I've been using this: def apply_binary (func, *args): if len (args) == 1: return args[0] elif len (args) == 2: return func (*args) else: return func ( apply_binary (func, *args[:len(args)/2]), apply_binary (func, *args[(len(args))/2:])) Then for example: punc2 = np.logical_and (u % 5 == 4, apply_binary (np.logical_or, u/5 == 3, u/5 == 8, u/5 == 13)) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On Tue, Jun 5, 2012 at 12:55 PM, mark florisson wrote: > It would be great if we implement the NEP listed above, but with a few > extensions. I think Numpy should handle the lazy evaluation part, and > determine when expressions should be evaluated, etc. However, for each > user operation, Numpy will call back a user-installed hook > implementing some interface, to allow various packages to provide > their own hooks to evaluate vector operations however they want. This > will include packages such as Theano, which could run things on the > GPU, Numexpr, and in the future > https://github.com/markflorisson88/minivect (which will likely have an > LLVM backend in the future, and possibly integrated with Numba to > allow inlining of numba ufuncs). The project above tries to bring > together all the different array expression compilers together in a > single framework, to provide efficient array expressions specialized > for any data layout (nditer on steroids if you will, with SIMD, > threaded and inlining capabilities). A global hook sounds ugly and hard to control -- it's hard to tell which operations should be deferred and which should be forced, etc. While it would be less magical, I think a more explicit API would in the end be easier to use... something like a, b, c, d = deferred([a, b, c, d]) e = a + b * c # 'e' is a deferred object too f = np.dot(e, d) # so is 'f' g = force(f) # 'g' is an ndarray # or force(f, out=g) But at that point, this could easily be an external library, right? All we'd need from numpy would be some way for external types to override the evaluation of ufuncs, np.dot, etc.? We've recently seen several reasons to want that functionality, and it seems like developing these "improved numexpr" ideas would be much easier if they didn't require doing deep surgery to numpy itself... -N ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1D array sorting ascending and descending by fields
On Mon, Jun 4, 2012 at 6:08 PM, Chris Barker wrote: > could you multiply the numeric field by -1, sort, then put it back > Yeah, that works great for my situation. Thanks Chris! On Mon, Jun 4, 2012 at 8:17 PM, Benjamin Root wrote: > While that may work for this users case, that would not work for all dtypes. > Some, such as timedelta, datetime and strings would not be able to be > multiplied by a number. > This is the reason why I thought there might be such a feature. > Would be an interesting feature to add, but I am not certain if the negative > sign notation would be best. Is it possible for a named field to start with > a negative sign? > I'm not sure about what is allowable in names, but I would be interested in getting involved with the NumPy project by helping to add this feature. I'll check out the contributing doc. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] varargs for logical_or, etc
On Tue, Jun 5, 2012 at 2:54 PM, Neal Becker wrote: > I think it's unfortunate that functions like logical_or are limited to binary. > > As a workaround, I've been using this: > > def apply_binary (func, *args): > if len (args) == 1: > return args[0] > elif len (args) == 2: > return func (*args) > else: > return func ( > apply_binary (func, *args[:len(args)/2]), > apply_binary (func, *args[(len(args))/2:])) > > Then for example: > > punc2 = np.logical_and (u % 5 == 4, > apply_binary (np.logical_or, u/5 == 3, u/5 == 8, u/5 == > 13)) reduce(np.logical_and, args) -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1D array sorting ascending and descending by fields
On Tue, Jun 5, 2012 at 1:17 AM, Benjamin Root wrote:
>
>
> On Monday, June 4, 2012, Chris Barker wrote:
>>
>> On Mon, Jun 4, 2012 at 11:10 AM, Patrick Redmond
>> wrote:
>> > Here's how I sorted primarily by field 'a' descending and secondarily by
>> > field 'b' ascending:
>>
>> could you multiply the numeric field by -1, sort, then put it back --
>> somethign like:
>>
>> data *- -1
>> data_sorted = np.sort(data, order=['a','b'])
>> data_sorted *= -1
>>
>> (reverse if necessary -- I lost track...)
>>
>> -Chris
>
>
>
> While that may work for this users case, that would not work for all dtypes.
> Some, such as timedelta, datetime and strings would not be able to be
> multiplied by a number.
>
> Would be an interesting feature to add, but I am not certain if the negative
> sign notation would be best. Is it possible for a named field to start with
> a negative sign?
Maybe add a reverse= argument (named after the corresponding argument
to list.sort and __builtins__.sorted).
# sorts in descending order, no fields required
np.sort([10, 20, 0], reverse=True)
# sorts in descending order
np.sort(rec_array, order=("a", "b"), reverse=True)
# ascending by "a" then descending by "b"
np.sort(rec_array, order=("a", "b"), reverse=(False, True))
?
-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] nditer_buffer_flag branch (was: Add data memory allocation tracing facilities. (#284))
On Tue, Jun 5, 2012 at 5:40 AM, Nathaniel Smith wrote: > On Tue, Jun 5, 2012 at 11:06 AM, Thouis (Ray) Jones wrote: > > All of the failing tests seem to have been caused by the buffer copy > bug, fixed in https://github.com/mwiebe/numpy/tree/nditer_buffer_flag(but > not yet pulled into numpy). > > > > I also have a version that implements tracing, with pure C in the > allocation functions writing to a dynamically allocated buffer, which must > then be fetched proactively by Python. However, I think this version is a > little nicer to use from the Python perspective. > > > > --- > > Reply to this email directly or view it on GitHub: > > https://github.com/numpy/numpy/pull/284#issuecomment-6121817 > > Speaking of which, Mark - what's the status of that nditer_buffer_flag > branch? Should there be a pull request? > Thanks for the nudge, I've made a PR. -Mark > > -N > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On 5 June 2012 14:58, Nathaniel Smith wrote:
> On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
> wrote:
>> It would be great if we implement the NEP listed above, but with a few
>> extensions. I think Numpy should handle the lazy evaluation part, and
>> determine when expressions should be evaluated, etc. However, for each
>> user operation, Numpy will call back a user-installed hook
>> implementing some interface, to allow various packages to provide
>> their own hooks to evaluate vector operations however they want. This
>> will include packages such as Theano, which could run things on the
>> GPU, Numexpr, and in the future
>> https://github.com/markflorisson88/minivect (which will likely have an
>> LLVM backend in the future, and possibly integrated with Numba to
>> allow inlining of numba ufuncs). The project above tries to bring
>> together all the different array expression compilers together in a
>> single framework, to provide efficient array expressions specialized
>> for any data layout (nditer on steroids if you will, with SIMD,
>> threaded and inlining capabilities).
>
> A global hook sounds ugly and hard to control -- it's hard to tell
> which operations should be deferred and which should be forced, etc.
Yes, but for the user the difference should not be visible (unless
operations can raise exceptions, in which case you choose the safe
path, or let the user configure what to do).
> While it would be less magical, I think a more explicit API would in
> the end be easier to use... something like
>
> a, b, c, d = deferred([a, b, c, d])
> e = a + b * c # 'e' is a deferred object too
> f = np.dot(e, d) # so is 'f'
> g = force(f) # 'g' is an ndarray
> # or
> force(f, out=g)
>
> But at that point, this could easily be an external library, right?
> All we'd need from numpy would be some way for external types to
> override the evaluation of ufuncs, np.dot, etc.? We've recently seen
> several reasons to want that functionality, and it seems like
> developing these "improved numexpr" ideas would be much easier if they
> didn't require doing deep surgery to numpy itself...
Definitely, but besides monkey-patch-chaining I think some
modifications would be required, but they would be reasonably simple.
Most of the functionality would be handled in one function, which most
ufuncs (the ones you care about, as well as ufunc (methods) like add)
call. E.g. if ((result = NPy_LazyEval("add", op1, op2)) return result;
, which is inserted after argument unpacking and sanity checking. You
could also do a per-module hook, and have the function look at
sys._getframe(1).f_globals, but that is fragile and won't work from C
or Cython code.
How did you have overrides in mind? I also found this thread:
http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
, but I think you want more than just to override ufuncs, you want
numpy to govern when stuff is allowed to be lazy and when stuff should
be evaluated (e.g. when it is indexed, slice assigned (although that
itself may also be lazy), etc). You don't want some funny object back
that doesn't work with things which are not overridden in numpy.
> -N
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Sun, Jun 3, 2012 at 12:04 PM, Ralf Gommers wrote: > > > On Sun, Jun 3, 2012 at 6:43 PM, Charles R Harris < > charlesr.har...@gmail.com> wrote: > >> Hi All, >> >> Numpy is approaching a time of transition. Ralf will be concentrating his >> efforts on Scipy > > > I'll write a separate post on that asap. > > >> and I will be cutting back on my work on Numpy. > > > I sincerely hope you don't cut back on your work too much Charles. You > have done an excellent job as "chief maintainer" over the last years. > > The 1.7 release looks to be delayed and I suspect that the Continuum >> Analytics folks will become increasingly dedicated to the big data push. We >> need new people to carry things forward and I think Nathaniel can pick up >> part of the load. >> > > Assuming he wants them, I am definitely +1 on giving Nathaniel commit > rights. His recent patches and debugging of issues were of high quality and > very helpful. > > OK, I went ahead and added him whether he wants it or not ;) Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Mon, Jun 4, 2012 at 10:12 PM, Dag Sverre Seljebotn
wrote:
> On 06/04/2012 09:06 PM, Mike Hansen wrote:
>> On Mon, May 28, 2012 at 3:15 AM, Mike Hansen wrote:
>>> In trying to upgrade NumPy within Sage, we notices some differences in
>>> behavior between 1.5 and 1.6. In particular, in 1.5, we have
>>>
>>> sage: f = 0.5
>>> sage: f.__array_interface__
>>> {'typestr': '=f8'}
>>> sage: numpy.array(f)
>>> array(0.5)
>>> sage: numpy.array(float(f))
>>> array(0.5)
>>>
>>> In 1.6, we get the following,
>>>
>>> sage: f = 0.5
>>> sage: f.__array_interface__
>>> {'typestr': '=f8'}
>>> sage: numpy.array(f)
>>> array(0.500, dtype=object)
>>>
>>> This seems to be do to the changes in PyArray_FromAny introduced in
>>> https://github.com/mwhansen/numpy/commit/2635398db3f26529ce2aaea4028a8118844f3c48
>>> . In particular, _array_find_type used to be used to query our
>>> __array_interface__ attribute, and it no longer seems to work. Is
>>> there a way to get the old behavior with the current code?
>
> No idea. If you want to spend the time to fix this properly, you could
> implement PEP 3118 and use that instead to export your array data (which
> can be done from Cython using __getbuffer__ on a Cython class).
I don't think that would work, because looking more closely, I don't
think they're actually doing anything like what
__array_interface__/PEP3118 are designed for. They just have some
custom class ("sage.rings.real_mpfr.RealLiteral", I guess an arbitrary
precision floating point of some sort?), and they want instances that
are passed to np.array() to be automatically coerced to another type
(float64) by default. But there's no buffer sharing or anything like
that going on at all. Mike, does that sound right?
This automagic coercion seems... in very dubious taste to me. (Why
does creating an array object imply that you want to throw away
precision? You can already throw away precision explicitly by doing
np.array(f, dtype=float).) But if this automatic coercion feature is
useful, then wouldn't it be better to have a different interface
instead of kluging it into __array_interface__, like we should check
for an attribute called __numpy_preferred_dtype__ or something?
-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 4:19 PM, Charles R Harris wrote: > > > On Sun, Jun 3, 2012 at 12:04 PM, Ralf Gommers > wrote: >> >> >> >> On Sun, Jun 3, 2012 at 6:43 PM, Charles R Harris >> wrote: >>> >>> Hi All, >>> >>> Numpy is approaching a time of transition. Ralf will be concentrating his >>> efforts on Scipy >> >> >> I'll write a separate post on that asap. >> >>> >>> and I will be cutting back on my work on Numpy. >> >> >> I sincerely hope you don't cut back on your work too much Charles. You >> have done an excellent job as "chief maintainer" over the last years. >> >>> The 1.7 release looks to be delayed and I suspect that the Continuum >>> Analytics folks will become increasingly dedicated to the big data push. We >>> need new people to carry things forward and I think Nathaniel can pick up >>> part of the load. >> >> >> Assuming he wants them, I am definitely +1 on giving Nathaniel commit >> rights. His recent patches and debugging of issues were of high quality and >> very helpful. >> > > OK, I went ahead and added him whether he wants it or not ;) Hah. Thanks! Is there a "committers guide" anywhere? By default I would assume that the rules are pretty much -- continue sending pull requests for my own changes (unless a trivial typo fix in a comment or something), go ahead and merge anyone else's pull request where things seem okay and my best judgement is we have consensus, fix things if my judgement was wrong? But I don't want to step on any toes... -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On Tue, Jun 5, 2012 at 4:12 PM, mark florisson
wrote:
> On 5 June 2012 14:58, Nathaniel Smith wrote:
>> On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
>> wrote:
>>> It would be great if we implement the NEP listed above, but with a few
>>> extensions. I think Numpy should handle the lazy evaluation part, and
>>> determine when expressions should be evaluated, etc. However, for each
>>> user operation, Numpy will call back a user-installed hook
>>> implementing some interface, to allow various packages to provide
>>> their own hooks to evaluate vector operations however they want. This
>>> will include packages such as Theano, which could run things on the
>>> GPU, Numexpr, and in the future
>>> https://github.com/markflorisson88/minivect (which will likely have an
>>> LLVM backend in the future, and possibly integrated with Numba to
>>> allow inlining of numba ufuncs). The project above tries to bring
>>> together all the different array expression compilers together in a
>>> single framework, to provide efficient array expressions specialized
>>> for any data layout (nditer on steroids if you will, with SIMD,
>>> threaded and inlining capabilities).
>>
>> A global hook sounds ugly and hard to control -- it's hard to tell
>> which operations should be deferred and which should be forced, etc.
>
> Yes, but for the user the difference should not be visible (unless
> operations can raise exceptions, in which case you choose the safe
> path, or let the user configure what to do).
>
>> While it would be less magical, I think a more explicit API would in
>> the end be easier to use... something like
>>
>> a, b, c, d = deferred([a, b, c, d])
>> e = a + b * c # 'e' is a deferred object too
>> f = np.dot(e, d) # so is 'f'
>> g = force(f) # 'g' is an ndarray
>> # or
>> force(f, out=g)
>>
>> But at that point, this could easily be an external library, right?
>> All we'd need from numpy would be some way for external types to
>> override the evaluation of ufuncs, np.dot, etc.? We've recently seen
>> several reasons to want that functionality, and it seems like
>> developing these "improved numexpr" ideas would be much easier if they
>> didn't require doing deep surgery to numpy itself...
>
> Definitely, but besides monkey-patch-chaining I think some
> modifications would be required, but they would be reasonably simple.
> Most of the functionality would be handled in one function, which most
> ufuncs (the ones you care about, as well as ufunc (methods) like add)
> call. E.g. if ((result = NPy_LazyEval("add", op1, op2)) return result;
> , which is inserted after argument unpacking and sanity checking. You
> could also do a per-module hook, and have the function look at
> sys._getframe(1).f_globals, but that is fragile and won't work from C
> or Cython code.
>
> How did you have overrides in mind?
My vague idea is that core numpy operations are about as fundamental
for scientific users as the Python builtin operations are, so they
should probably be overrideable in a similar way. So we'd teach numpy
functions to check for methods named like "__numpy_ufunc__" or
"__numpy_dot__" and let themselves be overridden if found. Like how
__gt__ and __add__ and stuff work. Or something along those lines.
> I also found this thread:
> http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
> , but I think you want more than just to override ufuncs, you want
> numpy to govern when stuff is allowed to be lazy and when stuff should
> be evaluated (e.g. when it is indexed, slice assigned (although that
> itself may also be lazy), etc). You don't want some funny object back
> that doesn't work with things which are not overridden in numpy.
My point is that probably numpy should *not* govern the decision about
what stuff should be lazy and what should be evaluated; that should be
governed by some combination of the user and
Numba/Theano/minivect/whatever. The toy API I sketched out would make
those decisions obvious and explicit. (And if the funny objects had an
__array_interface__ attribute that automatically forced evaluation
when accessed, then they'd work fine with code that was expecting an
array, or if they were assigned to a "real" ndarray, etc.)
-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1D array sorting ascending and descending by fields
On Tue, Jun 5, 2012 at 10:49 AM, Nathaniel Smith wrote:
> On Tue, Jun 5, 2012 at 1:17 AM, Benjamin Root wrote:
> >
> >
> > On Monday, June 4, 2012, Chris Barker wrote:
> >>
> >> On Mon, Jun 4, 2012 at 11:10 AM, Patrick Redmond
> >> wrote:
> >> > Here's how I sorted primarily by field 'a' descending and secondarily
> by
> >> > field 'b' ascending:
> >>
> >> could you multiply the numeric field by -1, sort, then put it back --
> >> somethign like:
> >>
> >> data *- -1
> >> data_sorted = np.sort(data, order=['a','b'])
> >> data_sorted *= -1
> >>
> >> (reverse if necessary -- I lost track...)
> >>
> >> -Chris
> >
> >
> >
> > While that may work for this users case, that would not work for all
> dtypes.
> > Some, such as timedelta, datetime and strings would not be able to be
> > multiplied by a number.
> >
> > Would be an interesting feature to add, but I am not certain if the
> negative
> > sign notation would be best. Is it possible for a named field to start
> with
> > a negative sign?
>
> Maybe add a reverse= argument (named after the corresponding argument
> to list.sort and __builtins__.sorted).
>
> # sorts in descending order, no fields required
> np.sort([10, 20, 0], reverse=True)
> # sorts in descending order
> np.sort(rec_array, order=("a", "b"), reverse=True)
> # ascending by "a" then descending by "b"
> np.sort(rec_array, order=("a", "b"), reverse=(False, True))
>
> ?
>
> -n
>
Clear, unambiguous, and works with the existing framework.
+1
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] varargs for logical_or, etc
On Tue, Jun 5, 2012 at 10:37 AM, Robert Kern wrote: > On Tue, Jun 5, 2012 at 2:54 PM, Neal Becker wrote: > > I think it's unfortunate that functions like logical_or are limited to > binary. > > > > As a workaround, I've been using this: > > > > def apply_binary (func, *args): > >if len (args) == 1: > >return args[0] > >elif len (args) == 2: > >return func (*args) > >else: > >return func ( > >apply_binary (func, *args[:len(args)/2]), > >apply_binary (func, *args[(len(args))/2:])) > > > > Then for example: > > > > punc2 = np.logical_and (u % 5 == 4, > > apply_binary (np.logical_or, u/5 == 3, u/5 == 8, > u/5 == > > 13)) > > > reduce(np.logical_and, args) > > I would love it if we could add something like that to the doc-string of those functions because I don't think it is immediately obvious. How do we do that for ufuncs? Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
Would lazy eval be able to eliminate temps in doing operations such as: np.sum (u != 23)? That is, now ops involving selecting elements of matrixes are often performed by first constructing temp matrixes, and the operating on them. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Tue, Jun 5, 2012 at 8:34 AM, Nathaniel Smith wrote:
> I don't think that would work, because looking more closely, I don't
> think they're actually doing anything like what
> __array_interface__/PEP3118 are designed for. They just have some
> custom class ("sage.rings.real_mpfr.RealLiteral", I guess an arbitrary
> precision floating point of some sort?), and they want instances that
> are passed to np.array() to be automatically coerced to another type
> (float64) by default. But there's no buffer sharing or anything like
> that going on at all. Mike, does that sound right?
Yes, there's no buffer sharing going on at all.
> This automagic coercion seems... in very dubious taste to me. (Why
> does creating an array object imply that you want to throw away
> precision?
The __array_interface__ attribute is a property which depends on the
precision of the ring. If it floats have enough precision, you just
get floats; otherwise you get objects.
> You can already throw away precision explicitly by doing
> np.array(f, dtype=float).) But if this automatic coercion feature is
> useful, then wouldn't it be better to have a different interface
> instead of kluging it into __array_interface__, like we should check
> for an attribute called __numpy_preferred_dtype__ or something?
It isn't just the array() calls which end up getting problems. For
example, in 1.5.x
sage: f = 10; type(f)
sage: numpy.arange(f)
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) #int64
while in 1.6.x
sage: numpy.arange(f)
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=object)
We also see problems with calls like
sage: scipy.stats.uniform(0,15).ppf([0.5,0.7])
array([ 7.5, 10.5])
which work in 1.5.x, but fail with a traceback "TypeError: array
cannot be safely cast to required type" in 1.6.x.
--Mike
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
> It isn't just the array() calls which end up getting problems. For > example, in 1.5.x > > sage: f = 10; type(f) > > sage: numpy.arange(f) > array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) #int64 > > while in 1.6.x > > sage: numpy.arange(f) > array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=object) > > We also see problems with calls like > > sage: scipy.stats.uniform(0,15).ppf([0.5,0.7]) > array([ 7.5, 10.5]) > > which work in 1.5.x, but fail with a traceback "TypeError: array > cannot be safely cast to required type" in 1.6.x. I'm getting problems like this after a 1.6 upgrade as well. Lots of object arrays being created when previously there would either be an error, or an array of floats. Also, lots of the "TypeError: array cannot be safely cast to required type" are cropping up. Honestly, most of these are in places where my code was lax and so I just cleaned things up to use the right dtypes etc. But still a bit unexpected in terms of having more code to fix than I was used to for 0.X numpy revisions. Just another data-point, though. Not really a complaint. Zach ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 10:25 AM, Nathaniel Smith wrote: > On Tue, Jun 5, 2012 at 4:19 PM, Charles R Harris > wrote: > > > > > > On Sun, Jun 3, 2012 at 12:04 PM, Ralf Gommers < > ralf.gomm...@googlemail.com> > > wrote: > >> > >> > >> > >> On Sun, Jun 3, 2012 at 6:43 PM, Charles R Harris > >> wrote: > >>> > >>> Hi All, > >>> > >>> Numpy is approaching a time of transition. Ralf will be concentrating > his > >>> efforts on Scipy > >> > >> > >> I'll write a separate post on that asap. > >> > >>> > >>> and I will be cutting back on my work on Numpy. > >> > >> > >> I sincerely hope you don't cut back on your work too much Charles. You > >> have done an excellent job as "chief maintainer" over the last years. > >> > >>> The 1.7 release looks to be delayed and I suspect that the Continuum > >>> Analytics folks will become increasingly dedicated to the big data > push. We > >>> need new people to carry things forward and I think Nathaniel can pick > up > >>> part of the load. > >> > >> > >> Assuming he wants them, I am definitely +1 on giving Nathaniel commit > >> rights. His recent patches and debugging of issues were of high quality > and > >> very helpful. > >> > > > > OK, I went ahead and added him whether he wants it or not ;) > > Hah. Thanks! > > Is there a "committers guide" anywhere? By default I would assume that > the rules are pretty much -- continue sending pull requests for my own > changes (unless a trivial typo fix in a comment or something), go > ahead and merge anyone else's pull request where things seem okay and > my best judgement is we have consensus, fix things if my judgement was > wrong? But I don't want to step on any toes... > > You can commit your own stuff also if someone signs off on it or it seems uncontroversial and has sat there for a while. It's mostly a judgement call. For the commits themselves, the github button doesn't do fast forward or whitespace cleanup, so I have the following alias in .git/config getpatch = !sh -c 'git co -b pull-$1 master &&\ curl https://github.com/numpy/nump/pull/$1.patch|\ git am -3 --whitespace=strip' - which opens a new branch pull-nnn and is useful for the bigger commits so they can be tested and then merged with master before pushing. The non-trivial commits should be tested with at least Python 2.4, 2.7, and 3.2. I also suggest running the one-file build for changes in core since most developers do the separate file thing and sometimes fail to catch single file build problems. Keep an eye on coding style, otherwise it will drift. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On 5 June 2012 18:21, Neal Becker wrote: > Would lazy eval be able to eliminate temps in doing operations such as: > > np.sum (u != 23)? > > That is, now ops involving selecting elements of matrixes are often performed > by > first constructing temp matrixes, and the operating on them. > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion Sure, yeah, it's pretty easy to generate a loop with an if statement and a reduction. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 11:52 AM, Charles R Harris wrote: > > > On Tue, Jun 5, 2012 at 10:25 AM, Nathaniel Smith wrote: > >> On Tue, Jun 5, 2012 at 4:19 PM, Charles R Harris >> wrote: >> > >> > >> > On Sun, Jun 3, 2012 at 12:04 PM, Ralf Gommers < >> ralf.gomm...@googlemail.com> >> > wrote: >> >> >> >> >> >> >> >> On Sun, Jun 3, 2012 at 6:43 PM, Charles R Harris >> >> wrote: >> >>> >> >>> Hi All, >> >>> >> >>> Numpy is approaching a time of transition. Ralf will be concentrating >> his >> >>> efforts on Scipy >> >> >> >> >> >> I'll write a separate post on that asap. >> >> >> >>> >> >>> and I will be cutting back on my work on Numpy. >> >> >> >> >> >> I sincerely hope you don't cut back on your work too much Charles. You >> >> have done an excellent job as "chief maintainer" over the last years. >> >> >> >>> The 1.7 release looks to be delayed and I suspect that the Continuum >> >>> Analytics folks will become increasingly dedicated to the big data >> push. We >> >>> need new people to carry things forward and I think Nathaniel can >> pick up >> >>> part of the load. >> >> >> >> >> >> Assuming he wants them, I am definitely +1 on giving Nathaniel commit >> >> rights. His recent patches and debugging of issues were of high >> quality and >> >> very helpful. >> >> >> > >> > OK, I went ahead and added him whether he wants it or not ;) >> >> Hah. Thanks! >> >> Is there a "committers guide" anywhere? By default I would assume that >> the rules are pretty much -- continue sending pull requests for my own >> changes (unless a trivial typo fix in a comment or something), go >> ahead and merge anyone else's pull request where things seem okay and >> my best judgement is we have consensus, fix things if my judgement was >> wrong? But I don't want to step on any toes... >> >> > You can commit your own stuff also if someone signs off on it or it seems > uncontroversial and has sat there for a while. It's mostly a judgement call. > > For the commits themselves, the github button doesn't do fast forward or > whitespace cleanup, so I have the following alias in .git/config > > getpatch = !sh -c 'git co -b pull-$1 master &&\ >curl https://github.com/numpy/nump/pull/$1.patch|\ >git am -3 --whitespace=strip' - > > which opens a new branch pull-nnn and is useful for the bigger commits so > they can be tested and then merged with master before pushing. The > non-trivial commits should be tested with at least Python 2.4, 2.7, and > 3.2. I also suggest running the one-file build for changes in core since > most developers do the separate file thing and sometimes fail to catch > single file build problems. > > Keep an eye on coding style, otherwise it will drift. > > And keep in mind that part of your job is to train new committers and help bring them up to speed. See yourself as a recruiter as well as a reviewer. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On 5 June 2012 17:38, Nathaniel Smith wrote:
> On Tue, Jun 5, 2012 at 4:12 PM, mark florisson
> wrote:
>> On 5 June 2012 14:58, Nathaniel Smith wrote:
>>> On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
>>> wrote:
It would be great if we implement the NEP listed above, but with a few
extensions. I think Numpy should handle the lazy evaluation part, and
determine when expressions should be evaluated, etc. However, for each
user operation, Numpy will call back a user-installed hook
implementing some interface, to allow various packages to provide
their own hooks to evaluate vector operations however they want. This
will include packages such as Theano, which could run things on the
GPU, Numexpr, and in the future
https://github.com/markflorisson88/minivect (which will likely have an
LLVM backend in the future, and possibly integrated with Numba to
allow inlining of numba ufuncs). The project above tries to bring
together all the different array expression compilers together in a
single framework, to provide efficient array expressions specialized
for any data layout (nditer on steroids if you will, with SIMD,
threaded and inlining capabilities).
>>>
>>> A global hook sounds ugly and hard to control -- it's hard to tell
>>> which operations should be deferred and which should be forced, etc.
>>
>> Yes, but for the user the difference should not be visible (unless
>> operations can raise exceptions, in which case you choose the safe
>> path, or let the user configure what to do).
>>
>>> While it would be less magical, I think a more explicit API would in
>>> the end be easier to use... something like
>>>
>>> a, b, c, d = deferred([a, b, c, d])
>>> e = a + b * c # 'e' is a deferred object too
>>> f = np.dot(e, d) # so is 'f'
>>> g = force(f) # 'g' is an ndarray
>>> # or
>>> force(f, out=g)
>>>
>>> But at that point, this could easily be an external library, right?
>>> All we'd need from numpy would be some way for external types to
>>> override the evaluation of ufuncs, np.dot, etc.? We've recently seen
>>> several reasons to want that functionality, and it seems like
>>> developing these "improved numexpr" ideas would be much easier if they
>>> didn't require doing deep surgery to numpy itself...
>>
>> Definitely, but besides monkey-patch-chaining I think some
>> modifications would be required, but they would be reasonably simple.
>> Most of the functionality would be handled in one function, which most
>> ufuncs (the ones you care about, as well as ufunc (methods) like add)
>> call. E.g. if ((result = NPy_LazyEval("add", op1, op2)) return result;
>> , which is inserted after argument unpacking and sanity checking. You
>> could also do a per-module hook, and have the function look at
>> sys._getframe(1).f_globals, but that is fragile and won't work from C
>> or Cython code.
>>
>> How did you have overrides in mind?
>
> My vague idea is that core numpy operations are about as fundamental
> for scientific users as the Python builtin operations are, so they
> should probably be overrideable in a similar way. So we'd teach numpy
> functions to check for methods named like "__numpy_ufunc__" or
> "__numpy_dot__" and let themselves be overridden if found. Like how
> __gt__ and __add__ and stuff work. Or something along those lines.
>
>> I also found this thread:
>> http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
>> , but I think you want more than just to override ufuncs, you want
>> numpy to govern when stuff is allowed to be lazy and when stuff should
>> be evaluated (e.g. when it is indexed, slice assigned (although that
>> itself may also be lazy), etc). You don't want some funny object back
>> that doesn't work with things which are not overridden in numpy.
>
> My point is that probably numpy should *not* govern the decision about
> what stuff should be lazy and what should be evaluated; that should be
> governed by some combination of the user and
> Numba/Theano/minivect/whatever. The toy API I sketched out would make
> those decisions obvious and explicit. (And if the funny objects had an
> __array_interface__ attribute that automatically forced evaluation
> when accessed, then they'd work fine with code that was expecting an
> array, or if they were assigned to a "real" ndarray, etc.)
That's disappointing though, since the performance drawbacks can
severely limit the usefulness for people with big data sets. Ideally,
you would take your intuitive numpy code, and make it go fast, without
jumping through hoops. Numpypy has lazy evaluation, I don't know how
good a job it does, but it does mean you can finally get fast numpy
code in an intuitive way (and even run it on a GPU if that is possible
and beneficial).
> -n
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
__
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Tue, Jun 5, 2012 at 11:51 AM, Zachary Pincus wrote: > > It isn't just the array() calls which end up getting problems. For > > example, in 1.5.x > > > > sage: f = 10; type(f) > > > > sage: numpy.arange(f) > > array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) #int64 > > > > while in 1.6.x > > > > sage: numpy.arange(f) > > array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=object) > > > > We also see problems with calls like > > > > sage: scipy.stats.uniform(0,15).ppf([0.5,0.7]) > > array([ 7.5, 10.5]) > > > > which work in 1.5.x, but fail with a traceback "TypeError: array > > cannot be safely cast to required type" in 1.6.x. > > I'm getting problems like this after a 1.6 upgrade as well. Lots of object > arrays being created when previously there would either be an error, or an > array of floats. > > Also, lots of the "TypeError: array cannot be safely cast to required > type" are cropping up. > > Honestly, most of these are in places where my code was lax and so I just > cleaned things up to use the right dtypes etc. But still a bit unexpected > in terms of having more code to fix than I was used to for 0.X numpy > revisions. > There is a fine line here. We do need to make people clean up lax code in order to improve numpy, but hopefully we can keep the cleanups reasonable. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
> There is a fine line here. We do need to make people clean up lax code in > order to improve numpy, but hopefully we can keep the cleanups reasonable. Oh agreed. Somehow, though, I was surprised by this, even though I keep tabs on the numpy lists -- at no point did it become clear that "big changes in how arrays get constructed and typecast are ahead that may require code fixes". That was my main point, but probably a PEBCAK issue more than anything. Zach ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Tue, Jun 5, 2012 at 8:41 PM, Zachary Pincus wrote: > > There is a fine line here. We do need to make people clean up lax code > in order to improve numpy, but hopefully we can keep the cleanups > reasonable. > > Oh agreed. Somehow, though, I was surprised by this, even though I keep > tabs on the numpy lists -- at no point did it become clear that "big > changes in how arrays get constructed and typecast are ahead that may > require code fixes". That was my main point, but probably a PEBCAK issue > more than anything. > It was fairly extensively discussed when introduced, http://thread.gmane.org/gmane.comp.python.numeric.general/44206, and again at some later point. Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] varargs for logical_or, etc
On Tue, Jun 5, 2012 at 6:59 PM, Benjamin Root wrote: > > > On Tue, Jun 5, 2012 at 10:37 AM, Robert Kern wrote: > >> On Tue, Jun 5, 2012 at 2:54 PM, Neal Becker wrote: >> > I think it's unfortunate that functions like logical_or are limited to >> binary. >> > >> > As a workaround, I've been using this: >> > >> > def apply_binary (func, *args): >> >if len (args) == 1: >> >return args[0] >> >elif len (args) == 2: >> >return func (*args) >> >else: >> >return func ( >> >apply_binary (func, *args[:len(args)/2]), >> >apply_binary (func, *args[(len(args))/2:])) >> > >> > Then for example: >> > >> > punc2 = np.logical_and (u % 5 == 4, >> > apply_binary (np.logical_or, u/5 == 3, u/5 == 8, >> u/5 == >> > 13)) >> >> >> reduce(np.logical_and, args) >> >> > I would love it if we could add something like that to the doc-string of > those functions because I don't think it is immediately obvious. How do we > do that for ufuncs? > Edit numpy/core/code_generators/ufunc_docstrings.py Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Tue, Jun 5, 2012 at 7:47 PM, Ralf Gommers wrote: > > > On Tue, Jun 5, 2012 at 8:41 PM, Zachary Pincus > wrote: >> >> > There is a fine line here. We do need to make people clean up lax code >> > in order to improve numpy, but hopefully we can keep the cleanups >> > reasonable. >> >> Oh agreed. Somehow, though, I was surprised by this, even though I keep >> tabs on the numpy lists -- at no point did it become clear that "big changes >> in how arrays get constructed and typecast are ahead that may require code >> fixes". That was my main point, but probably a PEBCAK issue more than >> anything. > > > It was fairly extensively discussed when introduced, > http://thread.gmane.org/gmane.comp.python.numeric.general/44206, and again > at some later point. Those are the not-yet-finalized changes in 1.7; Zachary (I think) is talking about problems upgrading from ~1.5 to 1.6. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
>> On Tue, Jun 5, 2012 at 8:41 PM, Zachary Pincus >> wrote: >>> There is a fine line here. We do need to make people clean up lax code in order to improve numpy, but hopefully we can keep the cleanups reasonable. >>> >>> Oh agreed. Somehow, though, I was surprised by this, even though I keep >>> tabs on the numpy lists -- at no point did it become clear that "big changes >>> in how arrays get constructed and typecast are ahead that may require code >>> fixes". That was my main point, but probably a PEBCAK issue more than >>> anything. >> >> >> It was fairly extensively discussed when introduced, >> http://thread.gmane.org/gmane.comp.python.numeric.general/44206, and again >> at some later point. > > Those are the not-yet-finalized changes in 1.7; Zachary (I think) is > talking about problems upgrading from ~1.5 to 1.6. Yes, unless I'm wrong I experienced these problems from 1.5.something to 1.6.1. I didn't take notes as it was in the middle of a deadline-crunch so I just fixed the code and moved on (long, stupid story about why the upgrade before a deadline...). It's just that the issues mentioned above seem to have hit me too and I wanted to mention that. But unhelpfully, I think, without code, and now I've hijacked this thread! Sorry. Zach ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 6:52 PM, Charles R Harris wrote: > > > On Tue, Jun 5, 2012 at 10:25 AM, Nathaniel Smith wrote: >> >> On Tue, Jun 5, 2012 at 4:19 PM, Charles R Harris >> wrote: >> > >> > >> > On Sun, Jun 3, 2012 at 12:04 PM, Ralf Gommers >> > >> > wrote: >> >> >> >> >> >> >> >> On Sun, Jun 3, 2012 at 6:43 PM, Charles R Harris >> >> wrote: >> >>> >> >>> Hi All, >> >>> >> >>> Numpy is approaching a time of transition. Ralf will be concentrating >> >>> his >> >>> efforts on Scipy >> >> >> >> >> >> I'll write a separate post on that asap. >> >> >> >>> >> >>> and I will be cutting back on my work on Numpy. >> >> >> >> >> >> I sincerely hope you don't cut back on your work too much Charles. You >> >> have done an excellent job as "chief maintainer" over the last years. >> >> >> >>> The 1.7 release looks to be delayed and I suspect that the Continuum >> >>> Analytics folks will become increasingly dedicated to the big data >> >>> push. We >> >>> need new people to carry things forward and I think Nathaniel can pick >> >>> up >> >>> part of the load. >> >> >> >> >> >> Assuming he wants them, I am definitely +1 on giving Nathaniel commit >> >> rights. His recent patches and debugging of issues were of high quality >> >> and >> >> very helpful. >> >> >> > >> > OK, I went ahead and added him whether he wants it or not ;) >> >> Hah. Thanks! >> >> Is there a "committers guide" anywhere? By default I would assume that >> the rules are pretty much -- continue sending pull requests for my own >> changes (unless a trivial typo fix in a comment or something), go >> ahead and merge anyone else's pull request where things seem okay and >> my best judgement is we have consensus, fix things if my judgement was >> wrong? But I don't want to step on any toes... >> > > You can commit your own stuff also if someone signs off on it or it seems > uncontroversial and has sat there for a while. It's mostly a judgement call. Speaking of which, this pull request has been sitting for a bit, waiting for your input :-) https://github.com/numpy/numpy/pull/280 > For the commits themselves, the github button doesn't do fast forward or > whitespace cleanup, so I have the following alias in .git/config > > getpatch = !sh -c 'git co -b pull-$1 master &&\ > curl https://github.com/numpy/nump/pull/$1.patch|\ > git am -3 --whitespace=strip' - > > which opens a new branch pull-nnn and is useful for the bigger commits so > they can be tested and then merged with master before pushing. The > non-trivial commits should be tested with at least Python 2.4, 2.7, and 3.2. > I also suggest running the one-file build for changes in core since most > developers do the separate file thing and sometimes fail to catch single > file build problems. Oops, heh. I don't know how to do the separate file thing, I've just been running single-file builds :-). > Keep an eye on coding style, otherwise it will drift. Thanks! -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On Tue, Jun 5, 2012 at 7:08 PM, mark florisson
wrote:
> On 5 June 2012 17:38, Nathaniel Smith wrote:
>> On Tue, Jun 5, 2012 at 4:12 PM, mark florisson
>> wrote:
>>> On 5 June 2012 14:58, Nathaniel Smith wrote:
On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
wrote:
> It would be great if we implement the NEP listed above, but with a few
> extensions. I think Numpy should handle the lazy evaluation part, and
> determine when expressions should be evaluated, etc. However, for each
> user operation, Numpy will call back a user-installed hook
> implementing some interface, to allow various packages to provide
> their own hooks to evaluate vector operations however they want. This
> will include packages such as Theano, which could run things on the
> GPU, Numexpr, and in the future
> https://github.com/markflorisson88/minivect (which will likely have an
> LLVM backend in the future, and possibly integrated with Numba to
> allow inlining of numba ufuncs). The project above tries to bring
> together all the different array expression compilers together in a
> single framework, to provide efficient array expressions specialized
> for any data layout (nditer on steroids if you will, with SIMD,
> threaded and inlining capabilities).
A global hook sounds ugly and hard to control -- it's hard to tell
which operations should be deferred and which should be forced, etc.
>>>
>>> Yes, but for the user the difference should not be visible (unless
>>> operations can raise exceptions, in which case you choose the safe
>>> path, or let the user configure what to do).
>>>
While it would be less magical, I think a more explicit API would in
the end be easier to use... something like
a, b, c, d = deferred([a, b, c, d])
e = a + b * c # 'e' is a deferred object too
f = np.dot(e, d) # so is 'f'
g = force(f) # 'g' is an ndarray
# or
force(f, out=g)
But at that point, this could easily be an external library, right?
All we'd need from numpy would be some way for external types to
override the evaluation of ufuncs, np.dot, etc.? We've recently seen
several reasons to want that functionality, and it seems like
developing these "improved numexpr" ideas would be much easier if they
didn't require doing deep surgery to numpy itself...
>>>
>>> Definitely, but besides monkey-patch-chaining I think some
>>> modifications would be required, but they would be reasonably simple.
>>> Most of the functionality would be handled in one function, which most
>>> ufuncs (the ones you care about, as well as ufunc (methods) like add)
>>> call. E.g. if ((result = NPy_LazyEval("add", op1, op2)) return result;
>>> , which is inserted after argument unpacking and sanity checking. You
>>> could also do a per-module hook, and have the function look at
>>> sys._getframe(1).f_globals, but that is fragile and won't work from C
>>> or Cython code.
>>>
>>> How did you have overrides in mind?
>>
>> My vague idea is that core numpy operations are about as fundamental
>> for scientific users as the Python builtin operations are, so they
>> should probably be overrideable in a similar way. So we'd teach numpy
>> functions to check for methods named like "__numpy_ufunc__" or
>> "__numpy_dot__" and let themselves be overridden if found. Like how
>> __gt__ and __add__ and stuff work. Or something along those lines.
>>
>>> I also found this thread:
>>> http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
>>> , but I think you want more than just to override ufuncs, you want
>>> numpy to govern when stuff is allowed to be lazy and when stuff should
>>> be evaluated (e.g. when it is indexed, slice assigned (although that
>>> itself may also be lazy), etc). You don't want some funny object back
>>> that doesn't work with things which are not overridden in numpy.
>>
>> My point is that probably numpy should *not* govern the decision about
>> what stuff should be lazy and what should be evaluated; that should be
>> governed by some combination of the user and
>> Numba/Theano/minivect/whatever. The toy API I sketched out would make
>> those decisions obvious and explicit. (And if the funny objects had an
>> __array_interface__ attribute that automatically forced evaluation
>> when accessed, then they'd work fine with code that was expecting an
>> array, or if they were assigned to a "real" ndarray, etc.)
>
> That's disappointing though, since the performance drawbacks can
> severely limit the usefulness for people with big data sets. Ideally,
> you would take your intuitive numpy code, and make it go fast, without
> jumping through hoops. Numpypy has lazy evaluation, I don't know how
> good a job it does, but it does mean you can finally get fast numpy
> code in an intuitive way (and even run it on a GPU if that is possible
> and beneficial).
All of these proposals require the user
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 1:14 PM, Nathaniel Smith wrote: > On Tue, Jun 5, 2012 at 6:52 PM, Charles R Harris > wrote: > > > > > > On Tue, Jun 5, 2012 at 10:25 AM, Nathaniel Smith wrote: > >> > >> On Tue, Jun 5, 2012 at 4:19 PM, Charles R Harris > >> wrote: > >> > > >> > > >> > On Sun, Jun 3, 2012 at 12:04 PM, Ralf Gommers > >> > > >> > wrote: > >> >> > >> >> > >> >> > >> >> On Sun, Jun 3, 2012 at 6:43 PM, Charles R Harris > >> >> wrote: > >> >>> > >> >>> Hi All, > >> >>> > >> >>> Numpy is approaching a time of transition. Ralf will be > concentrating > >> >>> his > >> >>> efforts on Scipy > >> >> > >> >> > >> >> I'll write a separate post on that asap. > >> >> > >> >>> > >> >>> and I will be cutting back on my work on Numpy. > >> >> > >> >> > >> >> I sincerely hope you don't cut back on your work too much Charles. > You > >> >> have done an excellent job as "chief maintainer" over the last years. > >> >> > >> >>> The 1.7 release looks to be delayed and I suspect that the Continuum > >> >>> Analytics folks will become increasingly dedicated to the big data > >> >>> push. We > >> >>> need new people to carry things forward and I think Nathaniel can > pick > >> >>> up > >> >>> part of the load. > >> >> > >> >> > >> >> Assuming he wants them, I am definitely +1 on giving Nathaniel commit > >> >> rights. His recent patches and debugging of issues were of high > quality > >> >> and > >> >> very helpful. > >> >> > >> > > >> > OK, I went ahead and added him whether he wants it or not ;) > >> > >> Hah. Thanks! > >> > >> Is there a "committers guide" anywhere? By default I would assume that > >> the rules are pretty much -- continue sending pull requests for my own > >> changes (unless a trivial typo fix in a comment or something), go > >> ahead and merge anyone else's pull request where things seem okay and > >> my best judgement is we have consensus, fix things if my judgement was > >> wrong? But I don't want to step on any toes... > >> > > > > You can commit your own stuff also if someone signs off on it or it seems > > uncontroversial and has sat there for a while. It's mostly a judgement > call. > > Speaking of which, this pull request has been sitting for a bit, > waiting for your input :-) > https://github.com/numpy/numpy/pull/280 > Mark and Travis made most of the comments so I figured it was up to them to sign off. If you think it is ready, go ahead and commit it, it's one of the reasons I gave you the premissions. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On 5 June 2012 20:17, Nathaniel Smith wrote:
> On Tue, Jun 5, 2012 at 7:08 PM, mark florisson
> wrote:
>> On 5 June 2012 17:38, Nathaniel Smith wrote:
>>> On Tue, Jun 5, 2012 at 4:12 PM, mark florisson
>>> wrote:
On 5 June 2012 14:58, Nathaniel Smith wrote:
> On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
> wrote:
>> It would be great if we implement the NEP listed above, but with a few
>> extensions. I think Numpy should handle the lazy evaluation part, and
>> determine when expressions should be evaluated, etc. However, for each
>> user operation, Numpy will call back a user-installed hook
>> implementing some interface, to allow various packages to provide
>> their own hooks to evaluate vector operations however they want. This
>> will include packages such as Theano, which could run things on the
>> GPU, Numexpr, and in the future
>> https://github.com/markflorisson88/minivect (which will likely have an
>> LLVM backend in the future, and possibly integrated with Numba to
>> allow inlining of numba ufuncs). The project above tries to bring
>> together all the different array expression compilers together in a
>> single framework, to provide efficient array expressions specialized
>> for any data layout (nditer on steroids if you will, with SIMD,
>> threaded and inlining capabilities).
>
> A global hook sounds ugly and hard to control -- it's hard to tell
> which operations should be deferred and which should be forced, etc.
Yes, but for the user the difference should not be visible (unless
operations can raise exceptions, in which case you choose the safe
path, or let the user configure what to do).
> While it would be less magical, I think a more explicit API would in
> the end be easier to use... something like
>
> a, b, c, d = deferred([a, b, c, d])
> e = a + b * c # 'e' is a deferred object too
> f = np.dot(e, d) # so is 'f'
> g = force(f) # 'g' is an ndarray
> # or
> force(f, out=g)
>
> But at that point, this could easily be an external library, right?
> All we'd need from numpy would be some way for external types to
> override the evaluation of ufuncs, np.dot, etc.? We've recently seen
> several reasons to want that functionality, and it seems like
> developing these "improved numexpr" ideas would be much easier if they
> didn't require doing deep surgery to numpy itself...
Definitely, but besides monkey-patch-chaining I think some
modifications would be required, but they would be reasonably simple.
Most of the functionality would be handled in one function, which most
ufuncs (the ones you care about, as well as ufunc (methods) like add)
call. E.g. if ((result = NPy_LazyEval("add", op1, op2)) return result;
, which is inserted after argument unpacking and sanity checking. You
could also do a per-module hook, and have the function look at
sys._getframe(1).f_globals, but that is fragile and won't work from C
or Cython code.
How did you have overrides in mind?
>>>
>>> My vague idea is that core numpy operations are about as fundamental
>>> for scientific users as the Python builtin operations are, so they
>>> should probably be overrideable in a similar way. So we'd teach numpy
>>> functions to check for methods named like "__numpy_ufunc__" or
>>> "__numpy_dot__" and let themselves be overridden if found. Like how
>>> __gt__ and __add__ and stuff work. Or something along those lines.
>>>
I also found this thread:
http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
, but I think you want more than just to override ufuncs, you want
numpy to govern when stuff is allowed to be lazy and when stuff should
be evaluated (e.g. when it is indexed, slice assigned (although that
itself may also be lazy), etc). You don't want some funny object back
that doesn't work with things which are not overridden in numpy.
>>>
>>> My point is that probably numpy should *not* govern the decision about
>>> what stuff should be lazy and what should be evaluated; that should be
>>> governed by some combination of the user and
>>> Numba/Theano/minivect/whatever. The toy API I sketched out would make
>>> those decisions obvious and explicit. (And if the funny objects had an
>>> __array_interface__ attribute that automatically forced evaluation
>>> when accessed, then they'd work fine with code that was expecting an
>>> array, or if they were assigned to a "real" ndarray, etc.)
>>
>> That's disappointing though, since the performance drawbacks can
>> severely limit the usefulness for people with big data sets. Ideally,
>> you would take your intuitive numpy code, and make it go fast, without
>> jumping through hoops. Numpypy has lazy evaluation, I don't know how
>> good a job it does, but it does mean you can finally get fast nu
Re: [Numpy-discussion] lazy evaluation
On Tue, Jun 5, 2012 at 9:47 PM, mark florisson
wrote:
> On 5 June 2012 20:17, Nathaniel Smith wrote:
>> On Tue, Jun 5, 2012 at 7:08 PM, mark florisson
>> wrote:
>>> On 5 June 2012 17:38, Nathaniel Smith wrote:
On Tue, Jun 5, 2012 at 4:12 PM, mark florisson
wrote:
> On 5 June 2012 14:58, Nathaniel Smith wrote:
>> On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
>> wrote:
>>> It would be great if we implement the NEP listed above, but with a few
>>> extensions. I think Numpy should handle the lazy evaluation part, and
>>> determine when expressions should be evaluated, etc. However, for each
>>> user operation, Numpy will call back a user-installed hook
>>> implementing some interface, to allow various packages to provide
>>> their own hooks to evaluate vector operations however they want. This
>>> will include packages such as Theano, which could run things on the
>>> GPU, Numexpr, and in the future
>>> https://github.com/markflorisson88/minivect (which will likely have an
>>> LLVM backend in the future, and possibly integrated with Numba to
>>> allow inlining of numba ufuncs). The project above tries to bring
>>> together all the different array expression compilers together in a
>>> single framework, to provide efficient array expressions specialized
>>> for any data layout (nditer on steroids if you will, with SIMD,
>>> threaded and inlining capabilities).
>>
>> A global hook sounds ugly and hard to control -- it's hard to tell
>> which operations should be deferred and which should be forced, etc.
>
> Yes, but for the user the difference should not be visible (unless
> operations can raise exceptions, in which case you choose the safe
> path, or let the user configure what to do).
>
>> While it would be less magical, I think a more explicit API would in
>> the end be easier to use... something like
>>
>> a, b, c, d = deferred([a, b, c, d])
>> e = a + b * c # 'e' is a deferred object too
>> f = np.dot(e, d) # so is 'f'
>> g = force(f) # 'g' is an ndarray
>> # or
>> force(f, out=g)
>>
>> But at that point, this could easily be an external library, right?
>> All we'd need from numpy would be some way for external types to
>> override the evaluation of ufuncs, np.dot, etc.? We've recently seen
>> several reasons to want that functionality, and it seems like
>> developing these "improved numexpr" ideas would be much easier if they
>> didn't require doing deep surgery to numpy itself...
>
> Definitely, but besides monkey-patch-chaining I think some
> modifications would be required, but they would be reasonably simple.
> Most of the functionality would be handled in one function, which most
> ufuncs (the ones you care about, as well as ufunc (methods) like add)
> call. E.g. if ((result = NPy_LazyEval("add", op1, op2)) return result;
> , which is inserted after argument unpacking and sanity checking. You
> could also do a per-module hook, and have the function look at
> sys._getframe(1).f_globals, but that is fragile and won't work from C
> or Cython code.
>
> How did you have overrides in mind?
My vague idea is that core numpy operations are about as fundamental
for scientific users as the Python builtin operations are, so they
should probably be overrideable in a similar way. So we'd teach numpy
functions to check for methods named like "__numpy_ufunc__" or
"__numpy_dot__" and let themselves be overridden if found. Like how
__gt__ and __add__ and stuff work. Or something along those lines.
> I also found this thread:
> http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
> , but I think you want more than just to override ufuncs, you want
> numpy to govern when stuff is allowed to be lazy and when stuff should
> be evaluated (e.g. when it is indexed, slice assigned (although that
> itself may also be lazy), etc). You don't want some funny object back
> that doesn't work with things which are not overridden in numpy.
My point is that probably numpy should *not* govern the decision about
what stuff should be lazy and what should be evaluated; that should be
governed by some combination of the user and
Numba/Theano/minivect/whatever. The toy API I sketched out would make
those decisions obvious and explicit. (And if the funny objects had an
__array_interface__ attribute that automatically forced evaluation
when accessed, then they'd work fine with code that was expecting an
array, or if they were assigned to a "real" ndarray, etc.)
>>>
>>> That's disappointing though, since the performance drawbacks can
>>> severely limit the usefulness for people with big data sets. Ideally,
>>> you would take your intuitive numpy code, and make it go fast, with
Re: [Numpy-discussion] lazy evaluation
On 06/05/2012 10:47 PM, mark florisson wrote:
> On 5 June 2012 20:17, Nathaniel Smith wrote:
>> On Tue, Jun 5, 2012 at 7:08 PM, mark florisson
>> wrote:
>>> On 5 June 2012 17:38, Nathaniel Smith wrote:
On Tue, Jun 5, 2012 at 4:12 PM, mark florisson
wrote:
> On 5 June 2012 14:58, Nathaniel Smith wrote:
>> On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
>> wrote:
>>> It would be great if we implement the NEP listed above, but with a few
>>> extensions. I think Numpy should handle the lazy evaluation part, and
>>> determine when expressions should be evaluated, etc. However, for each
>>> user operation, Numpy will call back a user-installed hook
>>> implementing some interface, to allow various packages to provide
>>> their own hooks to evaluate vector operations however they want. This
>>> will include packages such as Theano, which could run things on the
>>> GPU, Numexpr, and in the future
>>> https://github.com/markflorisson88/minivect (which will likely have an
>>> LLVM backend in the future, and possibly integrated with Numba to
>>> allow inlining of numba ufuncs). The project above tries to bring
>>> together all the different array expression compilers together in a
>>> single framework, to provide efficient array expressions specialized
>>> for any data layout (nditer on steroids if you will, with SIMD,
>>> threaded and inlining capabilities).
>>
>> A global hook sounds ugly and hard to control -- it's hard to tell
>> which operations should be deferred and which should be forced, etc.
>
> Yes, but for the user the difference should not be visible (unless
> operations can raise exceptions, in which case you choose the safe
> path, or let the user configure what to do).
>
>> While it would be less magical, I think a more explicit API would in
>> the end be easier to use... something like
>>
>> a, b, c, d = deferred([a, b, c, d])
>> e = a + b * c # 'e' is a deferred object too
>> f = np.dot(e, d) # so is 'f'
>> g = force(f) # 'g' is an ndarray
>> # or
>> force(f, out=g)
>>
>> But at that point, this could easily be an external library, right?
>> All we'd need from numpy would be some way for external types to
>> override the evaluation of ufuncs, np.dot, etc.? We've recently seen
>> several reasons to want that functionality, and it seems like
>> developing these "improved numexpr" ideas would be much easier if they
>> didn't require doing deep surgery to numpy itself...
>
> Definitely, but besides monkey-patch-chaining I think some
> modifications would be required, but they would be reasonably simple.
> Most of the functionality would be handled in one function, which most
> ufuncs (the ones you care about, as well as ufunc (methods) like add)
> call. E.g. if ((result = NPy_LazyEval("add", op1, op2)) return result;
> , which is inserted after argument unpacking and sanity checking. You
> could also do a per-module hook, and have the function look at
> sys._getframe(1).f_globals, but that is fragile and won't work from C
> or Cython code.
>
> How did you have overrides in mind?
My vague idea is that core numpy operations are about as fundamental
for scientific users as the Python builtin operations are, so they
should probably be overrideable in a similar way. So we'd teach numpy
functions to check for methods named like "__numpy_ufunc__" or
"__numpy_dot__" and let themselves be overridden if found. Like how
__gt__ and __add__ and stuff work. Or something along those lines.
> I also found this thread:
> http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
> , but I think you want more than just to override ufuncs, you want
> numpy to govern when stuff is allowed to be lazy and when stuff should
> be evaluated (e.g. when it is indexed, slice assigned (although that
> itself may also be lazy), etc). You don't want some funny object back
> that doesn't work with things which are not overridden in numpy.
My point is that probably numpy should *not* govern the decision about
what stuff should be lazy and what should be evaluated; that should be
governed by some combination of the user and
Numba/Theano/minivect/whatever. The toy API I sketched out would make
those decisions obvious and explicit. (And if the funny objects had an
__array_interface__ attribute that automatically forced evaluation
when accessed, then they'd work fine with code that was expecting an
array, or if they were assigned to a "real" ndarray, etc.)
>>>
>>> That's disappointing though, since the performance drawbacks can
>>> severely limit the usefulness for people with big data sets. Ideally,
>>> you would take your intuitive numpy code, and make it go fast, with
Re: [Numpy-discussion] lazy evaluation
On 5 June 2012 22:29, Nathaniel Smith wrote:
> On Tue, Jun 5, 2012 at 9:47 PM, mark florisson
> wrote:
>> On 5 June 2012 20:17, Nathaniel Smith wrote:
>>> On Tue, Jun 5, 2012 at 7:08 PM, mark florisson
>>> wrote:
On 5 June 2012 17:38, Nathaniel Smith wrote:
> On Tue, Jun 5, 2012 at 4:12 PM, mark florisson
> wrote:
>> On 5 June 2012 14:58, Nathaniel Smith wrote:
>>> On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
>>> wrote:
It would be great if we implement the NEP listed above, but with a few
extensions. I think Numpy should handle the lazy evaluation part, and
determine when expressions should be evaluated, etc. However, for each
user operation, Numpy will call back a user-installed hook
implementing some interface, to allow various packages to provide
their own hooks to evaluate vector operations however they want. This
will include packages such as Theano, which could run things on the
GPU, Numexpr, and in the future
https://github.com/markflorisson88/minivect (which will likely have an
LLVM backend in the future, and possibly integrated with Numba to
allow inlining of numba ufuncs). The project above tries to bring
together all the different array expression compilers together in a
single framework, to provide efficient array expressions specialized
for any data layout (nditer on steroids if you will, with SIMD,
threaded and inlining capabilities).
>>>
>>> A global hook sounds ugly and hard to control -- it's hard to tell
>>> which operations should be deferred and which should be forced, etc.
>>
>> Yes, but for the user the difference should not be visible (unless
>> operations can raise exceptions, in which case you choose the safe
>> path, or let the user configure what to do).
>>
>>> While it would be less magical, I think a more explicit API would in
>>> the end be easier to use... something like
>>>
>>> a, b, c, d = deferred([a, b, c, d])
>>> e = a + b * c # 'e' is a deferred object too
>>> f = np.dot(e, d) # so is 'f'
>>> g = force(f) # 'g' is an ndarray
>>> # or
>>> force(f, out=g)
>>>
>>> But at that point, this could easily be an external library, right?
>>> All we'd need from numpy would be some way for external types to
>>> override the evaluation of ufuncs, np.dot, etc.? We've recently seen
>>> several reasons to want that functionality, and it seems like
>>> developing these "improved numexpr" ideas would be much easier if they
>>> didn't require doing deep surgery to numpy itself...
>>
>> Definitely, but besides monkey-patch-chaining I think some
>> modifications would be required, but they would be reasonably simple.
>> Most of the functionality would be handled in one function, which most
>> ufuncs (the ones you care about, as well as ufunc (methods) like add)
>> call. E.g. if ((result = NPy_LazyEval("add", op1, op2)) return result;
>> , which is inserted after argument unpacking and sanity checking. You
>> could also do a per-module hook, and have the function look at
>> sys._getframe(1).f_globals, but that is fragile and won't work from C
>> or Cython code.
>>
>> How did you have overrides in mind?
>
> My vague idea is that core numpy operations are about as fundamental
> for scientific users as the Python builtin operations are, so they
> should probably be overrideable in a similar way. So we'd teach numpy
> functions to check for methods named like "__numpy_ufunc__" or
> "__numpy_dot__" and let themselves be overridden if found. Like how
> __gt__ and __add__ and stuff work. Or something along those lines.
>
>> I also found this thread:
>> http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
>> , but I think you want more than just to override ufuncs, you want
>> numpy to govern when stuff is allowed to be lazy and when stuff should
>> be evaluated (e.g. when it is indexed, slice assigned (although that
>> itself may also be lazy), etc). You don't want some funny object back
>> that doesn't work with things which are not overridden in numpy.
>
> My point is that probably numpy should *not* govern the decision about
> what stuff should be lazy and what should be evaluated; that should be
> governed by some combination of the user and
> Numba/Theano/minivect/whatever. The toy API I sketched out would make
> those decisions obvious and explicit. (And if the funny objects had an
> __array_interface__ attribute that automatically forced evaluation
> when accessed, then they'd work fine with code that was expecting an
> array, or if they were assigned to a "real" ndarray, etc.)
That's disappointing though, since the performance drawbacks can
seve
Re: [Numpy-discussion] lazy evaluation
On 5 June 2012 22:36, Dag Sverre Seljebotn wrote:
> On 06/05/2012 10:47 PM, mark florisson wrote:
>> On 5 June 2012 20:17, Nathaniel Smith wrote:
>>> On Tue, Jun 5, 2012 at 7:08 PM, mark florisson
>>> wrote:
On 5 June 2012 17:38, Nathaniel Smith wrote:
> On Tue, Jun 5, 2012 at 4:12 PM, mark florisson
> wrote:
>> On 5 June 2012 14:58, Nathaniel Smith wrote:
>>> On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
>>> wrote:
It would be great if we implement the NEP listed above, but with a few
extensions. I think Numpy should handle the lazy evaluation part, and
determine when expressions should be evaluated, etc. However, for each
user operation, Numpy will call back a user-installed hook
implementing some interface, to allow various packages to provide
their own hooks to evaluate vector operations however they want. This
will include packages such as Theano, which could run things on the
GPU, Numexpr, and in the future
https://github.com/markflorisson88/minivect (which will likely have an
LLVM backend in the future, and possibly integrated with Numba to
allow inlining of numba ufuncs). The project above tries to bring
together all the different array expression compilers together in a
single framework, to provide efficient array expressions specialized
for any data layout (nditer on steroids if you will, with SIMD,
threaded and inlining capabilities).
>>>
>>> A global hook sounds ugly and hard to control -- it's hard to tell
>>> which operations should be deferred and which should be forced, etc.
>>
>> Yes, but for the user the difference should not be visible (unless
>> operations can raise exceptions, in which case you choose the safe
>> path, or let the user configure what to do).
>>
>>> While it would be less magical, I think a more explicit API would in
>>> the end be easier to use... something like
>>>
>>> a, b, c, d = deferred([a, b, c, d])
>>> e = a + b * c # 'e' is a deferred object too
>>> f = np.dot(e, d) # so is 'f'
>>> g = force(f) # 'g' is an ndarray
>>> # or
>>> force(f, out=g)
>>>
>>> But at that point, this could easily be an external library, right?
>>> All we'd need from numpy would be some way for external types to
>>> override the evaluation of ufuncs, np.dot, etc.? We've recently seen
>>> several reasons to want that functionality, and it seems like
>>> developing these "improved numexpr" ideas would be much easier if they
>>> didn't require doing deep surgery to numpy itself...
>>
>> Definitely, but besides monkey-patch-chaining I think some
>> modifications would be required, but they would be reasonably simple.
>> Most of the functionality would be handled in one function, which most
>> ufuncs (the ones you care about, as well as ufunc (methods) like add)
>> call. E.g. if ((result = NPy_LazyEval("add", op1, op2)) return result;
>> , which is inserted after argument unpacking and sanity checking. You
>> could also do a per-module hook, and have the function look at
>> sys._getframe(1).f_globals, but that is fragile and won't work from C
>> or Cython code.
>>
>> How did you have overrides in mind?
>
> My vague idea is that core numpy operations are about as fundamental
> for scientific users as the Python builtin operations are, so they
> should probably be overrideable in a similar way. So we'd teach numpy
> functions to check for methods named like "__numpy_ufunc__" or
> "__numpy_dot__" and let themselves be overridden if found. Like how
> __gt__ and __add__ and stuff work. Or something along those lines.
>
>> I also found this thread:
>> http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
>> , but I think you want more than just to override ufuncs, you want
>> numpy to govern when stuff is allowed to be lazy and when stuff should
>> be evaluated (e.g. when it is indexed, slice assigned (although that
>> itself may also be lazy), etc). You don't want some funny object back
>> that doesn't work with things which are not overridden in numpy.
>
> My point is that probably numpy should *not* govern the decision about
> what stuff should be lazy and what should be evaluated; that should be
> governed by some combination of the user and
> Numba/Theano/minivect/whatever. The toy API I sketched out would make
> those decisions obvious and explicit. (And if the funny objects had an
> __array_interface__ attribute that automatically forced evaluation
> when accessed, then they'd work fine with code that was expecting an
> array, or if they were assigned to a "real" ndarray, etc.)
That's disappointing though, since the performance drawbacks can
s
Re: [Numpy-discussion] commit rights for Nathaniel
A couple of notes from the IPython workflow in case it's of use to you guys: On Tue, Jun 5, 2012 at 10:52 AM, Charles R Harris wrote: > > For the commits themselves, the github button doesn't do fast forward or > whitespace cleanup, so I have the following alias in .git/config > > getpatch = !sh -c 'git co -b pull-$1 master &&\ > curl https://github.com/numpy/nump/pull/$1.patch|\ > git am -3 --whitespace=strip' - > > which opens a new branch pull-nnn and is useful for the bigger commits so > they can be tested and then merged with master before pushing. The > non-trivial commits should be tested with at least Python 2.4, 2.7, and 3.2. > I also suggest running the one-file build for changes in core since most > developers do the separate file thing and sometimes fail to catch single > file build problems. 1) We've settled on using the green button rather than something like the above, because we decided that having the no-ff was actually a *good* thing (and yes, this reverses my initial opinion on the matter). The reasoning that convinced me was that the merge commit in itself is signal, not noise: - it indicates who did the final reviewing and merging (which doesn't happen in a ff merge b/c there's no separate merge commit) - it serves as a good place to cleanly summarize the PR itself, which could possibly contain many commits. It's the job and responsibility of the person doing the merge to understand the PR enough to explain it succinctly, so that one can read just that message and get a realistic idea of what the say 100 commits that went in were meant to do. These merge commits are the right thing to read when building release notes, instead of having to slog through the individual commits. - this way, the DAG's topology immediately shows what went in with review and what was committed without review (hopefully only small/trivial/emergency fixes). - even if the PR has a single commit, it's still OK to do this, as it marks the reviewer (and credits the reviewer as well, which is actual work). For all these reasons, I'm very happy that we reversed our policy and now *only* use the green button to merge, and *never* do a FF merge. We only commit directly to master in the case of absolutely trivial typo fixes or emergency 'my god master is borked' scenarios. 2) I'd encourage you to steal/improve our 'test_pr / post_pr_test' as well as git-mrb tools: https://github.com/ipython/ipython/blob/master/tools/test_pr.py https://github.com/ipython/ipython/blob/master/tools/post_pr_test.py https://github.com/ipython/ipython/blob/master/tools/git-mrb In particular test_pr is a *huge* help. We now almost never merge something that doesn't have a test_pr report. Here's an example where test_pr revealed initially problems, later fixed: https://github.com/ipython/ipython/pull/1847 Once the fix was confirmed, it was easy to merge. It routinely catches python3 errors we put in because most of the core devs don't use python3 regularly. But now I'm not worried about it anymore, as I know the problems will be caught before merging (I used to feel guilty for constantly breaking py3 and having poor Thomas Kluyver have to clean up my messes). Cheers, f ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 4:59 PM, Fernando Perez wrote: > A couple of notes from the IPython workflow in case it's of use to you > guys: > > On Tue, Jun 5, 2012 at 10:52 AM, Charles R Harris > wrote: > > > > For the commits themselves, the github button doesn't do fast forward or > > whitespace cleanup, so I have the following alias in .git/config > > > > getpatch = !sh -c 'git co -b pull-$1 master &&\ > >curl https://github.com/numpy/nump/pull/$1.patch|\ > >git am -3 --whitespace=strip' - > > > > which opens a new branch pull-nnn and is useful for the bigger commits so > > they can be tested and then merged with master before pushing. The > > non-trivial commits should be tested with at least Python 2.4, 2.7, and > 3.2. > > I also suggest running the one-file build for changes in core since most > > developers do the separate file thing and sometimes fail to catch single > > file build problems. > > 1) We've settled on using the green button rather than something like > the above, because we decided that having the no-ff was actually a > *good* thing (and yes, this reverses my initial opinion on the > matter). The reasoning that convinced me was that the merge commit in > itself is signal, not noise: > > - it indicates who did the final reviewing and merging (which doesn't > happen in a ff merge b/c there's no separate merge commit) > > - it serves as a good place to cleanly summarize the PR itself, which > could possibly contain many commits. It's the job and responsibility > of the person doing the merge to understand the PR enough to explain > it succinctly, so that one can read just that message and get a > realistic idea of what the say 100 commits that went in were meant to > do. These merge commits are the right thing to read when building > release notes, instead of having to slog through the individual > commits. > > - this way, the DAG's topology immediately shows what went in with > review and what was committed without review (hopefully only > small/trivial/emergency fixes). > > - even if the PR has a single commit, it's still OK to do this, as it > marks the reviewer (and credits the reviewer as well, which is actual > work). > > For all these reasons, I'm very happy that we reversed our policy and > now *only* use the green button to merge, and *never* do a FF merge. > We only commit directly to master in the case of absolutely trivial > typo fixes or emergency 'my god master is borked' scenarios. > > 2) I'd encourage you to steal/improve our 'test_pr / post_pr_test' > as well as git-mrb tools: > > https://github.com/ipython/ipython/blob/master/tools/test_pr.py > https://github.com/ipython/ipython/blob/master/tools/post_pr_test.py > https://github.com/ipython/ipython/blob/master/tools/git-mrb > > In particular test_pr is a *huge* help. We now almost never merge > something that doesn't have a test_pr report. Here's an example where > test_pr revealed initially problems, later fixed: > > https://github.com/ipython/ipython/pull/1847 > > Once the fix was confirmed, it was easy to merge. It routinely > catches python3 errors we put in because most of the core devs don't > use python3 regularly. But now I'm not worried about it anymore, as I > know the problems will be caught before merging (I used to feel guilty > for constantly breaking py3 and having poor Thomas Kluyver have to > clean up my messes). > > There are other advantages to pulling down the patch. Fixups can be merged together, commit comments enhanced, whitespace removed, style cleanups can be added, tests can be run, and the PR is automatically rebased. I still like fast forward for single commit merges, for larger merges I specify no-ff so that things come in as a well defined chunk. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 4:15 PM, Charles R Harris wrote: > There are other advantages to pulling down the patch. Fixups can be merged > together, commit comments enhanced, whitespace removed, style cleanups can > be added, tests can be run, and the PR is automatically rebased. I still > like fast forward for single commit merges, for larger merges I specify > no-ff so that things come in as a well defined chunk. Sure, that's a decision each project can take as it prefers: we've taken the approach that the person doing the merge does *not* massage the history as presented in the PR; instead we have submitters fix things up when deemed necessary (and we help them out a bit with git-fu if needed). And for single commit merges, we use the merge commit as topological evidence that there was review, which is very useful when looking retrospectively at the project. But each project must find how it best wants to proceed, I'm only offering our perspective in case any of it is useful for numpy. You guys will cherrypick the pieces that merge cleanly for numpy ;) Cheers, f ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy.clip behavior at max and min of dtypes
Can the following function be written using numpy.clip? In some other way? Does numpy.clip satisfy condition 4 below? Does numpy.clip satisfy some closely related condition? Define a function clipcast: output = clipcast(arr, dtype=None, out=None) 1. All arrays have int or float dtypes. 2. Exactly one of the keyword arguments "dtype" and "out" must be used. If "dtype" is given, then output has that dtype. 3. "output" has the same shape as "arr". 4. Let ER be the set of all the real numbers that can be exactly represented by the output dtype. ER is finite and bounded. Let themin = min(ER) and themax = max(ER). For any real number x, define a function f(x) by If x is in ER, define f(x) = x. If x is between two consecutive numbers, u and v, in ER, then define f(x) = u or f(x) = v. Probably the choice would be made using a C cast. If x < themin, define f(x) = themin. If x > themax, define f(x) = themax. If x is an element of arr, say Arr[I], then output[I] == f(x) where I is any index that defines a single element of arr. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
During the original discussion, Gael pointed out that the changes would probably break some code (which might need to be cleaned up but still). I think it was underestimated how quickly people would upgrade and see the changes and therefore be able to report problems. We are talking about a 1.7 release, but there are still people who have not upgraded their code to use 1.6 (when some of the big changes occurred). This should probably guide our view of how long it takes to migrate behavior in NumPy and minimize migration difficulties for users. -Travis On Jun 5, 2012, at 2:01 PM, Zachary Pincus wrote: >>> On Tue, Jun 5, 2012 at 8:41 PM, Zachary Pincus >>> wrote: > There is a fine line here. We do need to make people clean up lax code > in order to improve numpy, but hopefully we can keep the cleanups > reasonable. Oh agreed. Somehow, though, I was surprised by this, even though I keep tabs on the numpy lists -- at no point did it become clear that "big changes in how arrays get constructed and typecast are ahead that may require code fixes". That was my main point, but probably a PEBCAK issue more than anything. >>> >>> >>> It was fairly extensively discussed when introduced, >>> http://thread.gmane.org/gmane.comp.python.numeric.general/44206, and again >>> at some later point. >> >> Those are the not-yet-finalized changes in 1.7; Zachary (I think) is >> talking about problems upgrading from ~1.5 to 1.6. > > Yes, unless I'm wrong I experienced these problems from 1.5.something to > 1.6.1. I didn't take notes as it was in the middle of a deadline-crunch so I > just fixed the code and moved on (long, stupid story about why the upgrade > before a deadline...). It's just that the issues mentioned above seem to have > hit me too and I wanted to mention that. But unhelpfully, I think, without > code, and now I've hijacked this thread! Sorry. > > Zach > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
>
> I don't think that would work, because looking more closely, I don't
> think they're actually doing anything like what
> __array_interface__/PEP3118 are designed for. They just have some
> custom class ("sage.rings.real_mpfr.RealLiteral", I guess an arbitrary
> precision floating point of some sort?), and they want instances that
> are passed to np.array() to be automatically coerced to another type
> (float64) by default. But there's no buffer sharing or anything like
> that going on at all. Mike, does that sound right?
>
> This automagic coercion seems... in very dubious taste to me. (Why
> does creating an array object imply that you want to throw away
> precision? You can already throw away precision explicitly by doing
> np.array(f, dtype=float).) But if this automatic coercion feature is
> useful, then wouldn't it be better to have a different interface
> instead of kluging it into __array_interface__, like we should check
> for an attribute called __numpy_preferred_dtype__ or something?
Interesting. It does look like off-label use of the __array_interface__
attribute. Given that "array" used to query the __array_interface__ attribute
for type discovery, I still wonder why it was disabled in 1.6?
-Travis
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Wed, Jun 6, 2012 at 5:11 AM, Travis Oliphant wrote: > During the original discussion, Gael pointed out that the changes would > probably break some code (which might need to be cleaned up but still). I > think it was underestimated how quickly people would upgrade and see the > changes and therefore be able to report problems. > > You're making the same mistake I made above. This error occurs in 1.6.x, so before the proposed change to casting='same_kind'. That's not actually the default right now by the way, in both 1.6.2 and current master the default is 'safe'. In [3]: np.__version__ Out[3]: '1.7.0.dev-fd78546' In [4]: print np.can_cast.__doc__ can_cast(from, totype, casting = 'safe') Ralf We are talking about a 1.7 release, but there are still people who have not > upgraded their code to use 1.6 (when some of the big changes occurred). > > This should probably guide our view of how long it takes to migrate > behavior in NumPy and minimize migration difficulties for users. > > > -Travis > > > > On Jun 5, 2012, at 2:01 PM, Zachary Pincus wrote: > > >>> On Tue, Jun 5, 2012 at 8:41 PM, Zachary Pincus < > zachary.pin...@yale.edu> > >>> wrote: > > > There is a fine line here. We do need to make people clean up lax > code > > in order to improve numpy, but hopefully we can keep the cleanups > > reasonable. > > Oh agreed. Somehow, though, I was surprised by this, even though I > keep > tabs on the numpy lists -- at no point did it become clear that "big > changes > in how arrays get constructed and typecast are ahead that may require > code > fixes". That was my main point, but probably a PEBCAK issue more than > anything. > >>> > >>> > >>> It was fairly extensively discussed when introduced, > >>> http://thread.gmane.org/gmane.comp.python.numeric.general/44206, and > again > >>> at some later point. > >> > >> Those are the not-yet-finalized changes in 1.7; Zachary (I think) is > >> talking about problems upgrading from ~1.5 to 1.6. > > > > Yes, unless I'm wrong I experienced these problems from 1.5.something to > 1.6.1. I didn't take notes as it was in the middle of a deadline-crunch so > I just fixed the code and moved on (long, stupid story about why the > upgrade before a deadline...). It's just that the issues mentioned above > seem to have hit me too and I wanted to mention that. But unhelpfully, I > think, without code, and now I've hijacked this thread! Sorry. > > > > Zach > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
