On Thu, 3 Nov 2005, Paul Cochrane wrote:
> On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
>
>> On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <[EMAIL PROTECTED]>
>> wrote:
>>
>>> I've got an application that I'm writing that autogenerates python
>>> code which I then execute with exec(). I know that this is not the
>>> best way to run things, and I'm not 100% sure as to what I really
>>> should do. I've had a look through Programming Python and the Python
>>> Cookbook, which have given me ideas, but nothing has gelled yet, so I
>>> thought I'd put the question to the community. But first, let me be a
>>> little more detailed in what I want to do:
Paul, this is a rather interesting problem. There are two aspects to it,
which i believe are probably separable: getting instructions from the
client to the server, and getting data back from the server to the client.
The former is more complex, i think, and what's attracted the attention so
far.
The first thing i'd say is that, while eval/exec is definitely a code
smell, that doesn't mean it's never the right solution. If you need to be
able to express complex things, python code might well be the best way to
do it, and the best way to evaluate python code is eval/exec.
>> It's a little hard to tell without knowing more about your user input
>> (command language?) syntax that is translated to or feeds the process
>> that "autogenerates python code".
>
> It's basically just a command language I guess.
Hang on - the stuff that the user writes is what you're calling "pyvisi
code", is that right? That doesn't look like 'just a command language',
that looks like python, using a library you've written. Or is there
another language, the "just a command language", on top of that?
And what you call "vtk-python code" - this is python again, but using the
renderer's native library, right?
And you generate the vtk-python from the pyvisi-python by executing the
pyvisi-python, there being (pluggable renderer-specific) logic in the guts
of your pyvisi classes to emit the vtk-python code, right? You're not
parsing anything?
>> There are lots of easy things you could do without generating and exec-ing
>> python code per se.
>
> I'd love to know of other options. I like the idea of generating the
> code one would have to write for a particular renderer so that if the
> user wanted to, they could use the autogenerated code to form the basis
> of a specific visualisation script they could then hack themselves.
If you want vtk-python code as an intermediate, i think you're stuck with
eval/exec [1].
> One of the main ideas of the module is to distill the common visualisation
> tasks down to a simple set of commands, and then let the interface work out
> how to actually implement that.
Okay. There's a classic design pattern called Interpreter that applies
here. This is one of the more complex patterns, and one that's rather
poorly explained in the Gang of Four book, so it's not well-known.
Basically, the idea is that you provide classes which make it possible for
a program to build structures encoding a series of instructions -
essentially, you define a language whose concrete syntax is objects, not
text - then you write code which takes such structures and carries out the
instructions encoded in them - an interpreter, in other words.
For example, here's a very simple example for doing basic arithmetic:
# the language
class expression(object):
pass
class constant(expression):
def __init__(self, value):
self.value = value
class unary(expression):
def __init__(self, op, arg):
self.op = op
self.arg = arg
class binary(expression):
def __init__(self, op, arg_l, arg_r):
self.op = op
self.arg_l = arg_l
self.arg_r = arg_r
# the interpreter
UNARY_OPS = {
"-": lambda x: -x,
"|": lambda x: abs(x) # apologies for abnormal syntax
}
BINARY_OPS = {
"+": lambda l, r: l + r,
"-": lambda l, r: l - r,
"*": lambda l, r: l * r,
"/": lambda l, r: l / r,
}
def evaluate(expr):
if isinstance(expr, constant):
return expr.value
elif isinstance(expr, unary):
op = UNARY_OPS[expr.op]
arg = evaluate(expr.arg)
return op(arg)
elif isinstance(expr, binary):
op = BINARY_OPS[expr.op]
arg_l = evaluate(expr.arg_l)
arg_r = evaluate(expr.arg_r)
return op(arg_l, arg_r)
else:
raise Exception, "unknown expression type: " + str(type(expr))
# a quick demo
expr = binary("-",
binary("*",
constant(2.0),
constant(3.0)),
unary("-",
binary("/",
constant(4.0),
constant(5.0))))
print evaluate(expr)
This is by no means a useful or well-designed bit of code, and there are
several things that could have been done differently (bare vs wrapped
constants, operations defined by a symbol vs expression subtypes for each
operation, etc), but i hope it gets the idea across - representing a
language using an object graph, which lets you write programs that can
speak that language.
Your code is already doing something a bit like this - you build scene
graphs, then call render on them to get them to do something. Instead of
that, you'd pass the whole scene to a renderer object, which would do the
rendering (directly, rather than by generating code). The point is that
the renderer could be in another process, provided you have a way to move
the scene graph from one process to another - the pickle module, for
example, or a custom serialisation format if you feel like reinventing the
wheel.
An approach like this has a natural solution to your second problem, too -
the evaluator function can return objects, which again can just be pickled
and sent over the network.
tom
[1] Okay, so there is a way to do this without ever actually creating
python code. You're not going to like this.
You need to apply the interpreter pattern to python itself. Well, a
simplified subset of it. Looking at your generated vtk-python code, you
basically do the following things:
- call methods with variables and literals as arguments
- throwing away the result
- or keeping it in a variable
- do for loops over ranges of integers
To make things a bit simpler, i'm going to add:
- getting attributes
- subscripting arrays
- return a value
You also need to do some arithmetic, i think; i leave that as an exercise
for the reader.
So we need a language like:
class statement(object):
pass
class invoke(statement):
def __init__(self, var, target, method, args):
self.var = var # name of variable for result; None to throw away
self.target = target
self.method = method
self.args = args
class get(statement):
def __init__(self, var, target, field):
self.var = var
self.target = target
self.field = field
class subscript(statement):
def __init__(self, var, target, index):
self.var = var
self.target = target
self.index = index
class forloop(statement):
def __init__(self, var, limit, body):
self.var = var
self.limit = limit
self.body = body # tuple of statements
def return_(statement):
def __init__(self, value):
self.value = value
With which we can write a script like:
vtk_script = [
invoke("_plot", "vtk", "vtkXYPlotActor", ()),
invoke("_renderer", "vtk", "vtkRenderer", ()),
invoke("_renderWindow", "vtk", "vtkRenderWindow", ()),
invoke(None, "_renderWindow", "AddRenderer", ("_renderer")),
invoke(None, "_renderWindow", "SetSize", (640, 480)),
invoke(None, "_renderer", "SetBackground", (1, 1, 1)),
invoke("_x", None, "array", ([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0,
8.0, 9.0],)),
get("vtk_vtkDataArray", "vtk", "vtkDataArray"),
get("vtk_VTK_FLOAT", "vtk", "VTK_FLOAT"),
invoke("_xData", "vtk_vtkDataArray", "CreateDataArray",
("vtk_VTK_FLOAT",)),
invoke("_x_len", None, "len", ("_x",)),
invoke(None, "_xData", "SetNumberOfTuples", ("_x_len",)),
invoke("_y0", None, "array", ([0.0, 1.0, 4.0, 9.0, 16.0, 25.0, 36.0,
49.0, 64.0, 81.0],)),
invoke("_y0Data", "vtk_vtkDataArray", "CreateDataArray",
("vtk_VTK_FLOAT",)),
invoke("_y0_len", None, "len", ("_y0",)),
invoke(None, "_y0Data", "SetNumberOfTuples", ("_y0_len",)),
forloop("i", "_x_len", (
subscript("_x_i", "_x", "i"),
invoke(None, "_xData", "SetTuple1", ("i", "_x_i")),)),
# etc
]
Or rather, your pyvisi classes can generate structures like this, exactly
as they currently generate code.
Code to convert this into python source is trivial, so i'll gloss over
that. The interpreter looks like this:
def isiterable(x):
return hasattr(x, "__iter__")
def execute(script, vars=None):
if (vars == None):
vars = {}
# initialise variables with 'vtk' and anything else you need
def decode_arg(arg):
if (isinstance(arg, str)):
return vars[arg]
elif (isiterable(arg)):
return map(decode_arg, arg)
else:
return arg
for stmt in script:
if (isinstance(stmt, invoke)):
target = vars[stmt.target]
method = getattr(target, stmt.method)
args = decode_arg(stmt.args)
result = method(*args)
if (stmt.var != None):
vars[stmt.var] = result
elif (isinstance(stmt, get)):
target = vars[stmt.target]
vars[stmt.var] = getattr(target, stmt.field)
elif (isinstance(stmt, subscript)):
target = vars[stmt.target]
vars[stmt.var] = getattr(target, decode_arg(stmt.index))
elif (isinstance(stmt, forloop)):
var = stmt.var
limit = decode_arg(stmt.limit)
body = stmt.body
for i in range(limit):
vars[var] = i
execute(body, vars)
elif (isinstance(stmt, return_)):
return vars[stmt.value]
# nb won't work from inside a for loop!
# you can use an exception to handle returns properly
Note that i haven't tested this, i've just written it off the top of my
head, so it probably won't work, but maybe you get the idea.
To be honest, i'd go with exec.
--
Fitter, Happier, More Productive.
--
http://mail.python.org/mailman/listinfo/python-list
