Re: compiler package vs parser
Kay Schluehr wrote: I realize that I probably ought to be trying this out with the newer ast stuff, but currently I am supporting code back to 2.3 and there's not much hope of doing it right there without using the compiler package. You might consider using the *builtin* parser module and forget about the compiler package if it is broken ( I take your word that it is ) or modern ast representations which aren't really necessary for Python anyway. import parser tree = parser.suite("def foo():\n print 42}\n") code = tree.compile() exec code foo() 42 This is also not 100% reliable ( at least not for all statements in all Python versions ) but it uses the internal parser/compiler and not a standard library compiler package that might not be that well maintained. ... thinking about it I just made the wrong decision back in 2004; we observed a semantic change caused by the new scoping rules and tried to fix using the wrong model; back then we were probably supporting 2.0 as well so the parser module probably wasn't available everywhere anyway; even today the ast stuff isn't available in 2.4. I prefer the ast approach as preppy is effectively indentation free which makes the tree harder to synthesize for the parser tree. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
> I realize that I probably ought to be trying this out with the newer ast > stuff, > but currently I am supporting code back to 2.3 and there's not much hope of > doing it right there without using the compiler package. You might consider using the *builtin* parser module and forget about the compiler package if it is broken ( I take your word that it is ) or modern ast representations which aren't really necessary for Python anyway. >>> import parser >>> tree = parser.suite("def foo():\n print 42}\n") >>> code = tree.compile() >>> exec code >>> foo() 42 This is also not 100% reliable ( at least not for all statements in all Python versions ) but it uses the internal parser/compiler and not a standard library compiler package that might not be that well maintained. -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
En Fri, 17 Apr 2009 10:55:46 -0300, Scott David Daniels escribió: Robin Becker wrote: def func(D): for k in D: exec '%s=D[%r]' % (k,k) print i, j, k print locals() print i, j, k if __name__=='__main__': func(dict(i=1,j=33)) end p.py the compiler package ends up treating i & j as global, whereas the modern analysis doesn't (and doesn't say they're definitely local either). If they are not definitely local, they are non-local. Locals are determined at function definition time, not function execution time. So, your expectations about what the exec statement can do above are mistaken. You may try to work your way around it, but, IMHO, you will not succeed. If the code above were to work as you wish, every access to every non-local in code that contains an "exec" would have to check a "new locals" dictionary just in case the exec added a local. And that's what happens. In absence of an exec statement, locals are optimized: the compiler knows exactly how many of them exist, and its names, just by static code analysis. But when the function contains an exec statement (or an "import *" statement) this is not possible anymore, and the compiler has to switch to another strategy and generate code using a less-optimized approach. Unknown variables are accessed using LOAD_NAME/STORE_NAME (require a name lookup) instead of the normal LOAD_FAST/STORE_FAST for local variables and LOAD_GLOBAL/STORE_GLOBAL for global ones. Think about what this code would have to do: > i = j = 42 > def func(D): > print i, j, k > for k in D: > exec '%s=D[%r]' % (k,k) > print i, j, k I don't completely understand what you wanted to show, but try this: py> i = j = 42 py> def f(): ... print i, j ... exec "k=1" ... print i, j, k ... exec "i=5" ... print i, j, k ... py> f() 42 42 42 42 1 5 42 1 py> i 42 py> j 42 py> k Traceback (most recent call last): File "", line 1, in NameError: name 'k' is not defined Note that i starts as a global name and after the exec "i=5" it becomes a local variable; and k is always a local variable even if there is no explicit assignment to it (except in the exec). -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
Robin Becker wrote: If I have messed that up then there should be some easy fix, otherwise if pycodegen is somehow not getting the semantics of the the variables i,j correct is there some way I can fix that def func(D): for k in D: exec '%s=D[%r]' % (k,k) print i, j, k print locals() print i, j, k if __name__=='__main__': func(dict(i=1,j=33)) end p.py the compiler package ends up treating i & j as global, whereas the modern analysis doesn't (and doesn't say they're definitely local either). If they are not definitely local, they are non-local. Locals are determined at function definition time, not function execution time. So, your expectations about what the exec statement can do above are mistaken. You may try to work your way around it, but, IMHO, you will not succeed. If the code above were to work as you wish, every access to every non-local in code that contains an "exec" would have to check a "new locals" dictionary just in case the exec added a local. Think about what this code would have to do: > i = j = 42 > def func(D): > print i, j, k > for k in D: > exec '%s=D[%r]' % (k,k) > print i, j, k --Scott David Daniels scott.dani...@acm.org -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
In article , Robin Becker wrote: > >My analysis of the problem is that in > > start p.py >def func(D): > for k in D: > exec '%s=D[%r]' % (k,k) > print i, j, k > print locals() > print i, j, k > >if __name__=='__main__': > func(dict(i=1,j=33)) > end p.py > >the compiler package ends up treating i & j as global, whereas the >modern analysis doesn't (and doesn't say they're definitely local >either). Looking at the code in Python/compile.c the compile_nameop >code seems to check for scopes FREE, CELL, LOCAL, GLOBAL_IMPLICIT & >GLOBAL_EXPLICIT whereas pycodegen.CodeGenerator._nameOp seems not to >know about GLOBAL_IMPLICIT/EXPLICIT but has only a single GLOBAL scope. At this point, given the lack of response from people who actually know the compiler internals, I think it's fair of you to try going to python-dev -- normally questions like this are off-topic, and if someone complains, I'll take the blame. ;-) -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ "If you think it's expensive to hire a professional to do the job, wait until you hire an amateur." --Red Adair -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
Kay Schluehr wrote: On 16 Apr., 11:41, Robin Becker wrote: Is the compiler package actually supposed to be equivalent to the parser module? No. The parser module creates a concrete parse tree ( CST ) whereas the compiler package transforms this CST into an AST for subsequent computations. In more recent versions those CST -> AST transformations are performed by the runtime and the Python compiler uses those internally produced ASTs. The Python 2.6 API to ASTs is the ast module. .. OK having digested most of what Kay said (my simple engineer's brain being rather stretched at the best of times) I would like to rephrase my question. Is it intended that the compile package be able to create the same code as the normal parser chain. I think I understand now that the compiler used internally is now exposed in various ways as built in modules. The example I gave earlier seems to indicate that the compiler package may not be creating the same final bytecode at least using the compilation approach that I used (cbased on the compiler package itself). If I have messed that up then there should be some easy fix, otherwise if pycodegen is somehow not getting the semantics of the the variables i,j correct is there some way I can fix that. I realize that I probably ought to be trying this out with the newer ast stuff, but currently I am supporting code back to 2.3 and there's not much hope of doing it right there without using the compiler package. My analysis of the problem is that in start p.py def func(D): for k in D: exec '%s=D[%r]' % (k,k) print i, j, k print locals() print i, j, k if __name__=='__main__': func(dict(i=1,j=33)) end p.py the compiler package ends up treating i & j as global, whereas the modern analysis doesn't (and doesn't say they're definitely local either). Looking at the code in Python/compile.c the compile_nameop code seems to check for scopes FREE, CELL, LOCAL, GLOBAL_IMPLICIT & GLOBAL_EXPLICIT whereas pycodegen.CodeGenerator._nameOp seems not to know about GLOBAL_IMPLICIT/EXPLICIT but has only a single GLOBAL scope. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
Kay Schluehr wrote: On 16 Apr., 11:41, Robin Becker wrote: Is the compiler package actually supposed to be equivalent to the parser module? No. The parser module creates a concrete parse tree ( CST ) whereas the compiler package transforms this CST into an AST for subsequent computations. In more recent versions those CST -> AST transformations are performed by the runtime and the Python compiler uses those internally produced ASTs. The Python 2.6 API to ASTs is the ast module. OK so I've been using the old compiler package to synthesize my trees, but presumably there are equivalents in the ast module so I could re-code my stuff to use the new module fairly easily. Presumably compiling the ast trees should produce identical code. On the other hand it seems as though there are at least three generations of AST lying about with compiler, ast (and _ast in 2.5) so multi version coding will be hard. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
On 16 Apr., 11:41, Robin Becker wrote: > Is the compiler package actually supposed to be equivalent to the parser > module? No. The parser module creates a concrete parse tree ( CST ) whereas the compiler package transforms this CST into an AST for subsequent computations. In more recent versions those CST -> AST transformations are performed by the runtime and the Python compiler uses those internally produced ASTs. The Python 2.6 API to ASTs is the ast module. -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
Robin Becker wrote: Aahz wrote: In article , Robin Becker wrote: Is the compiler package actually supposed to be equivalent to the parser module? Before I poke my nose into this, what versions of Python have you tried? I'm using 2.6. I just checked and it's the same in 2.5. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
Aahz wrote: In article , Robin Becker wrote: Is the compiler package actually supposed to be equivalent to the parser module? Before I poke my nose into this, what versions of Python have you tried? I'm using 2.6. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
In article , Robin Becker wrote: > >Is the compiler package actually supposed to be equivalent to the >parser module? Before I poke my nose into this, what versions of Python have you tried? -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ "If you think it's expensive to hire a professional to do the job, wait until you hire an amateur." --Red Adair -- http://mail.python.org/mailman/listinfo/python-list
compiler package vs parser
Is the compiler package actually supposed to be equivalent to the parser module? I ask because the following code start p.py def func(D): for k in D: exec '%s=D[%r]' % (k,k) print i, j, k print locals() print i, j, k if __name__=='__main__': func(dict(i=1,j=33)) end p.py when run through the compiler package to produce a module has different code to that produced by the standard compilation (detected by dis.dis). In particular the variables i and j in func above are treated differently; in the standard compiler case LOAD_NAME is used and in the code from the package LOAD_GLOBAL is used. The code used to create the synthetic module is start tp.py from compiler import parse, pycodegen, misc, syntax import time, struct, marshal txt=open('p.py','r').read() tree=parse(txt) print 'tree\n',tree def _get_tree(tree,filename): misc.set_filename(filename, tree) syntax.check(tree) return tree def getPycHeader(): mtime = time.time() mtime = struct.pack('The module synp.pyc fails with a traceback (as expected because there are no global i,j), but p.py runs OK. I assume that my attempt to compile the tree is broken (is missing some special traverse etc) otherwise the code would end up the same (except for line numbering which I have ignored). -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list