Francesc Alted writes: > On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote: >> You need at least a slightly different Python API to get anywhere, so >> numexpr/Theano is the right place to work on an implementation of this >> idea. Of course it would be nice if numexpr/Theano offered something as >> convenient as >> >> with lazy: >> arr = A + B + C # with all of these NumPy arrays >> # compute upon exiting…
> Hmm, that would be cute indeed. Do you have an idea on how the code in the > with > context could be passed to the Python AST compiler (à la numexpr.evaluate("A > + B > + C"))? Well, I started writing some experiments to "almost transparently" translate regular ndarray operations to numexpr strings (or others) using only python code. The concept is very simple: # you only need the first one to start building the AST a = lazy(np.arange(16)) b = np.arange(16) res = a + b + 3 print evaluate(res) # the actual evaluation can be delayed to something like __repr__ or __str__ print repr(res) print res # you could also delay evaluation until someone uses res to create a new array My target was to use this to also generate optimized GPU kernels in-flight using pycuda, but I think some other relatively recent project already performed something similar (w.r.t. generating cuda kernels out of python expressions). The supporting code for numexpr was something like: import numexpr import numpy as np def build_arg_expr (arg, args): if isinstance(arg, Expr): # recursively build the expression arg_expr, arg_args = arg.build_expr() args.update(arg_args) return arg_expr else: # unique argument identifier arg_id = "arg_%d" % id(arg) args[arg_id] = arg return arg_id # generic expression builder class Expr: def evaluate(self): expr, args = self.build_expr() return numexpr.evaluate(expr, local_dict = args, global_dict = {}) def __repr__ (self): return self.evaluate().__repr__() def __str__ (self): return self.evaluate().__str__() def __add__ (self, other): return ExprAdd(self, other) # expression builder for adds class ExprAdd(Expr): def __init__(self, arg1, arg2): self.arg1 = arg1 self.arg2 = arg2 def build_expr(self): args = {} expr1 = build_arg_expr(self.arg1, args) expr2 = build_arg_expr(self.arg2, args) return "("+expr1+") + ("+expr2+")", args # ndarray-like class to generate expression builders class LazyNdArray(np.ndarray): def __add__ (self, other): return ExprAdd(self, other) # build a LazyNdArray def lazy (arg): return arg.view(LazyNdArray) # evaluate with numexpr an arbitrary expression builder def evaluate(arg): return arg.evaluate() The thing here is to always return to the user something that looks like an ndarray. As you can see the whole thing is not very complex, but some less funny code had to be written meanwhile for work and I just dropped this :) Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion